Maya works in operations and Leon works in HR. They can ask the same Northstar assistant about leave, but they should not necessarily receive the same evidence. Maya may read the general policy and her own balance. Leon's HR role permits a defined set of employee records. Similar wording in a question does not create identical access rights.
Maya asks, “What adjustments did Leon make to Priya's leave record?” The application should deny access or provide a neutral response under company policy. It should not disclose a document title, snippet or citation that reveals the restricted adjustment. Asking the model to suppress sensitive content after it has already received it is too late for the model boundary.
A good authorization test is deliberately boring. Put a unique synthetic marker in a document Maya cannot access, then inspect the model input, logs, cache and citations. The marker should never appear on her path. Repeat the test after revoking access and after an HR user has asked about the same topic. That is where permission-blind caches and stale group membership show up.
Deletion and permission changes need the same end-to-end thinking. Removing a source file may leave copies in chunks, embeddings, backups or cached answers. Decide which stores retain content, which deletion rules apply and how quickly changed permissions must take effect. Treat embeddings and retrieval metadata as potentially sensitive even when they are not readable prose.
Diagnose retrieval before changing the model
If a retrieval answer is wrong, I would inspect the source version and the passages returned before changing the model. Was the exception split into another chunk? Did exact terminology defeat semantic search? Did permission filtering leave too little evidence? Approximate indexes can also trade completeness for speed, so measure the candidates the system actually returned. [12]
Only after that would I inspect generation. Check whether each material claim follows from the supplied evidence, whether citations resolve to an authorized source, and whether the answer acknowledges missing or conflicting information. A citation beside a made-up statement does not make the statement grounded.
For Northstar, a useful first retrieval evaluation includes ordinary policy questions, an outdated policy, an exception in an adjacent section, an unanswerable question and multiple user roles. Each exposes a different failure. One overall satisfaction score would hide most of them.