Production RAG: What Actually Works Past the Demo

RAG demos are easy. You take a set of documents, chunk them, embed them, store the embeddings in a vector database, and wire the retrieval step to an LLM. The demo works. The questions you test it with get answered correctly. You ship it.

Then users start asking questions you did not test, and the system answers them confidently and incorrectly. Or it retrieves the right document but the wrong chunk, and the answer is subtly wrong in a way that is hard to catch on inspection. Or it works well for six months and silently degrades as the document corpus grows and the retrieval precision drops.

This is the gap between a RAG demo and a RAG product. The architecture is not the problem — retrieval-augmented generation is a well-understood approach with a clear theoretical basis. The problem is a series of implementation decisions that look fine in a demo context and break down under real usage: chunking that destroys context, embeddings that miss exact-match queries, no reranking, no citation enforcement, and no defined behaviour for when the answer is not in the corpus.

The Chunking Problem

The most common cause of RAG failure is chunking. Documents are typically split by a naive rule — every N tokens, every paragraph break, every newline — without considering what context those chunks carry when retrieved independently of the documents they came from.

A clause in a contract that references "the party defined in section 2.1" is meaningless when retrieved on its own. A bullet point from a policy document that begins "Additionally..." gives the model nothing to anchor the answer to. A technical specification that repeats a table across pages creates duplicate chunks that confuse the retrieval signal. A chunk that starts mid-sentence because the previous chunk ended at a character limit may not be coherent enough for the model to reason about.

The fix is document-type-aware chunking. Contracts chunk differently from policy documents, which chunk differently from technical specifications, which chunk differently from support articles. The chunk size, the overlap, and the metadata attached to each chunk all need to match the structure of the document type they are cutting.

For most production RAG systems this means:

• Chunking at natural semantic boundaries — section breaks, numbered items, question-answer pairs — rather than at fixed token counts that cut across logical units.
• Storing surrounding context as metadata on every chunk: the section heading, the document title, the page number, the subsection. When this metadata is included in the context window alongside the chunk, the model has the reference frame it needs to interpret the content correctly.
• For documents with repeated structure — tables, forms, structured lists — extracting structured data instead of treating them as prose. A table that says "rate: 3.75%, term: 36 months" is better represented as key-value pairs than as a tokenised string of whitespace-separated numbers.
• Overlapping chunks, with a window of 10 to 20 percent, so that information at chunk boundaries is not lost because the split happened to land in the middle of a relevant sentence.

The Retrieval Problem

Dense vector search is not sufficient on its own. Vector embeddings are trained to capture semantic similarity — they are good at finding paraphrases and topically related content, and they are poor at exact-match queries for specific identifiers, product codes, clause numbers, or names.

If a user asks "what is the rate under clause 4.2(b)?" and the document says "clause 4.2(b): the applicable rate is 3.75%," a dense-only retrieval may not return that chunk because the embedding of the query and the embedding of the answer do not score above the retrieval threshold. The query contains a specific label — "clause 4.2(b)" — that dense retrieval treats as just another token sequence rather than as a unique identifier that must match exactly.

The fix is hybrid search: dense vector retrieval combined with keyword retrieval (BM25 or a similar sparse method), with the results merged using reciprocal rank fusion or a learned reranker. Dense retrieval finds conceptually related chunks. Sparse retrieval finds exact- match terms, part numbers, clause references, and names. Together they cover what either approach misses alone.

This is not a novel technique. Hybrid search has been standard practice in information retrieval research for years and consistently outperforms dense-only retrieval on domain- specific corpora where users query for specific identifiers alongside semantic concepts. Most teams implement it later than they should because the initial demo only tests semantic queries chosen to showcase the embedding quality.

The Reranking Problem

Retrieving the relevant chunk is necessary but not sufficient. The top-k results from your retrieval step will include the right answer — assuming retrieval recall is high — but may not place it in the first position. The model reads the context window sequentially, and relevant chunks placed at position 7 of 10 will have less influence on the generated answer than less-relevant chunks in the first position.

Reranking is a second pass over the retrieved chunks, using a cross-encoder model that scores each chunk against the specific query. Unlike a bi-encoder embedding model (which encodes the query and the chunk independently), a cross-encoder sees both together and produces a relevance score that accounts for their interaction. This is slower — it requires one inference per chunk rather than one similarity search — but it reliably improves precision on the final answer because it brings the most query-specific chunks to the top.

For document intelligence applications — legal documents, contracts, policy manuals, technical specifications — reranking is not an optional optimisation. It is the step that takes retrieval from "probably has the answer in context" to "has the answer at the top of context where the model will use it." The gains are measurable: run a test set of question-answer pairs from your documents and compare top-1 precision with and without reranking. The improvement is typically large enough to be immediately obvious.

The Hallucination Problem

Vector retrieval guarantees that similar content exists somewhere in your corpus. It does not guarantee that the retrieved chunk actually answers the query. When the gap is present, the model will fill it from its training weights, and it will do so confidently and fluently. The result is an answer that sounds authoritative and is wrong in a way that is difficult to detect without already knowing the correct answer.

Two constraints that consistently reduce hallucination in production RAG systems:

Grounding prompts. The system prompt should explicitly instruct the model to answer only from the provided context, to state when it cannot find the answer rather than inferring or extrapolating, and to distinguish between what the documents say and what the model might otherwise believe. This is not a complete solution — a model that has learned a fact during pretraining will sometimes surface it regardless of the grounding instruction — but it substantially reduces confabulation on domain-specific queries where the model has less confident prior beliefs.

Citation enforcement. Require the model to cite the specific source — document name, page number, section reference — for every factual claim in its answer. This serves two purposes. It forces the model to anchor its answer to retrieved content, because fabricating a specific citation is harder than fabricating a general claim. And it gives the user a path to verify the answer independently, which is what makes the system trustworthy rather than just convenient.

For high-stakes document queries — contract review, regulatory compliance, policy interpretation — add a confidence threshold: if no retrieved chunk scores above a minimum relevance threshold, return a "not found" response instead of asking the model to answer. This is better than a hallucinated answer with a fabricated citation, and it tells you which queries your corpus does not yet cover — which is actionable information for improving coverage.

The Evaluation Problem

You cannot know whether your RAG system is working without measuring it. The absence of obvious failures is not evidence of correctness. A system that hallucinations on 10% of queries will pass informal testing because the test queries were chosen to be answerable, not to expose failure modes.

Build a test set before you tune anything. Fifty to a hundred question-answer pairs, sampled from the actual documents users will query, where the correct answer is a specific passage from a specific document. This is the instrument that lets you measure the effect of every change you make to the pipeline: the chunking strategy, the retrieval parameters, the reranker, the prompt.

Measure three things on this test set:

• Retrieval recall at k: is the correct chunk in the top k results? This measures whether the retrieval step is capable of finding the right answer, independently of whether the model uses it correctly.
• Answer correctness: does the generated answer match the ground truth? This is the end-to-end metric, but it conflates retrieval quality and generation quality. When it drops, retrieval recall tells you which half of the pipeline to fix.
• Citation accuracy: does the citation point to the correct document and section? A system with high answer correctness and low citation accuracy is fabricating citations that happen to support correct answers — which is better than a system that fabricates wrong answers, but not trustworthy.

Run this evaluation on every meaningful change to the pipeline. When answer correctness drops, retrieval recall tells you whether the problem is in retrieval or generation. When retrieval recall drops after adding new documents, you know the new documents require chunking changes or corpus-specific tuning. Without the evaluation set, you are guessing.

What Production-Ready Actually Looks Like

A RAG system that holds up in production has:

• Document-type-aware chunking with surrounding context stored as metadata on every chunk, and structured extraction for tabular and form-based content.
• Hybrid retrieval — dense embeddings plus sparse keyword search — with results merged by reciprocal rank fusion before the reranking step.
• A cross-encoder reranking pass that brings the most query-specific chunks to the top of the context window.
• A system prompt that grounds answers in retrieved content and requires citations on every factual claim.
• A relevance threshold below which the system returns "not found" rather than asking the model to answer from insufficient context.
• A test set of 50 to 100 evaluated question-answer pairs, run on every change to the pipeline, with retrieval recall and answer correctness tracked over time.

None of this is exotic or expensive. All of it is the difference between a RAG demo and a RAG product — the difference between a system that works when you demonstrate it and a system that your users trust to give them correct answers when you are not watching.