A retrieval-augmented generation demo takes an afternoon: split a PDF into chunks, embed them, put the top five in a prompt. The same system in production starts returning the wrong policy, missing the newest document, or leaking a paragraph the user should not see. The model is rarely the problem. Retrieval is.
Chunk by structure, not by character count
Fixed-size chunks cut sentences and tables in half, then the embedding represents neither idea well. Split along the document's structure first: headings, sections, list items, table rows. Keep a small overlap only where a section genuinely crosses a boundary. Store the heading path with each chunk so you can show the model where the text came from.
document: employee-handbook.pdf
section: Leave policy > Parental leave
chunk: eligibility, duration, notice period
metadata: {section_path, updated_at, access_group}Hybrid retrieval beats pure vectors
Embeddings are good at semantic similarity and bad at exact identifiers, product codes, names and rare terms. A keyword index such as BM25 catches those. Run both, merge the candidates, then rerank with a cross-encoder or a small model. The reranker is often the highest-leverage upgrade in the whole pipeline.
Do not skip the boring filters. Date, source, language and access group should be applied before candidates reach the model. A permission check after generation is too late.
- Vector search for meaning, BM25 for exact terms.
- Merge candidates, then rerank the top 30–50.
- Filter by tenant, access group and freshness before reranking.
- Keep the source URL and heading path for citations.
Freshness is an indexing problem
Most 'the bot is outdated' reports are not model failures. A document changed and the index did not. Track a content hash and an updated timestamp per source. Re-index incrementally on change, and keep a deletion path so removed documents stop appearing. A full nightly rebuild is fine for a small corpus; it is not fine when the corpus is large or changes hourly.
Evaluate retrieval separately from generation
If you only measure the final answer, you cannot tell whether the retriever missed the right document or the model ignored it. Build a small retrieval set: a question and the document IDs that should be retrieved. Measure recall at k and mean reciprocal rank. Then measure answer faithfulness against the retrieved context.
Twenty to fifty curated questions will expose more than a generic benchmark. Include the ugly cases: acronyms, ambiguous names, a question whose answer spans two documents, and a question with no answer in the corpus.
If the right document is not in the context, no prompt can save the answer.
When RAG is the wrong tool
Not every question needs retrieval. If the answer is a single row in a database, query the database. If the whole corpus fits comfortably in the context window and changes rarely, pass it directly and skip the index. If the task is classification or extraction, fine-tuning or a deterministic parser may be cheaper and more accurate.
RAG is for open-ended questions over a corpus that is too large, too fresh or too permission-sensitive to paste into a prompt. Use it for that, not as a default architecture.
What to remember
- Chunk along document structure and keep the heading path as metadata.
- Combine vector search with keyword search and rerank the merged candidates.
- Apply tenant, permission and freshness filters before the model sees anything.
- Measure retrieval recall separately from answer quality.
- Skip RAG when a SQL query, a small fixed context or a parser solves the problem.
