The demo-to-production gap
A RAG prototype over a hundred clean documents will look excellent. The same architecture over four hundred thousand documents of mixed quality, with overlapping versions and inconsistent structure, will not — and the failure mode is confident, well-written, wrong answers.
Nearly all of that degradation happens before the model is involved. If retrieval returns the wrong chunks, no amount of prompt engineering recovers the answer.
Retrieval is the engineering problem
- Chunking that respects document structure — headings and sections, not fixed character counts
- Hybrid search combining vector similarity with keyword matching for identifiers and part numbers
- Metadata filtering so a query can be scoped to jurisdiction, product line or document version
- Reranking the candidate set before it reaches the model's context
- Explicit handling of superseded documents, which is where most wrong answers originate
Evaluate before you ship
Build a labelled question set from real user queries — a hundred is enough to be useful. Measure retrieval precision separately from answer quality, because they fail independently and the fixes are different. Re-run the suite on every chunking, embedding or prompt change.
Then instrument production: log the retrieved chunks alongside every answer. When someone reports a bad response, the retrieved set almost always explains it in seconds.