- Document permissions must be enforced during retrieval, not after generation.
- Retrieval quality needs its own evaluation set, separate from answer quality.
- Citations without a verified source binding are decoration, not evidence.
Model the corpus before choosing a model
Most RAG failures begin in the connector and chunking layer. Source systems have different freshness, deletion semantics and access rules. A shared drive, a ticketing system and a contract repository do not share the same permission model. We map every source into a governed document contract with stable identifiers, version, checksum, classification and ACL metadata. Deletion propagation is designed at ingestion time, because a stale embedding is a data-retention incident waiting to happen.
Chunking is a retrieval decision, not a formatting decision. Contract clauses, API references and support tickets need different boundaries. We test chunk sizes against real queries, preserve parent-child context and keep a pointer to the exact source span so citations can be validated.
- Connector-specific sync and deletion policy
- Stable document, version and checksum identities
- Classification and ACL metadata on every chunk
- Parent-child context for precise citations
Retrieve inside the permission boundary
Hybrid retrieval combines lexical recall with dense similarity, then applies metadata filters before context assembly. The critical rule is that permission filters run inside the retrieval query. Generating an answer from a document the user cannot open and then hiding the citation is not access control. For high-sensitivity corpora we assemble context per request and cache only within a tenant and permission scope.
A reranker improves precision, but it cannot repair missing evidence. We measure recall@k against a labelled query set, track zero-result and low-confidence rates, and route weak retrievals to a clarifying response instead of a confident hallucination.
- Hybrid BM25 and dense retrieval
- ACL filtering before reranking
- Query-set recall@k and zero-result monitoring
- Confidence thresholds with escalation paths
Evaluate the system, not the prompt
A prompt change is a release. We maintain golden question sets with expected sources, answer constraints and adversarial cases. The evaluation harness scores retrieval recall, citation validity, groundedness, refusal behaviour and latency. Tool calls are treated as privileged actions with explicit allowlists, argument validation and human approval where the consequence is material.
In production, traces connect the user question, retrieved chunks, model version, prompt version and final answer. That trace is what makes regressions diagnosable. Without it, teams debate prompts while the real failure sits in an index that stopped syncing three weeks ago.
- Golden sets for retrieval and answer quality
- Citation validation against source spans
- Tool permissions and human approval gates
- Versioned prompts, models and indexes with rollback
Questions we hear in the room
Can we keep all data inside our cloud?
Yes. The retrieval service, vector store, inference runtime and traces can run inside your VPC or on dedicated group infrastructure, subject to model licensing and capacity.
Do we need fine-tuning?
Usually not at the start. Retrieval quality, permissions and evaluation produce more reliable gains. Fine-tuning is justified only when a measured task gap remains.
