Design a RAG system over a customer's 40,000 internal documents (policies, contracts, SOPs). Where does retrieval quality actually come from?
Answer it out loud first, then compare. That's what makes it stick.
Past the embed-store-top-k tutorial, quality comes from five places. Parsing and structure-aware chunking, because no chunk size fixes a bad parse. Metadata on every chunk. Hybrid retrieval for recall, then a reranker for precision at the top. Query handling, like rewriting follow-ups. And measurement: I score retrieval and answers separately on real questions. I'd start with one document set and one team, always cite sources, and measure before adding anything fancier.
- Clarify users, doc types, change rate, permissions
- Parse and chunk by the document's own structure
- Metadata on every chunk (version, section, ACL)
- Hybrid retrieval then rerank
- Cited generation, measured per layer
Interviewers want you past the tutorial pipeline (embed, store, top-k, generate) and into the decisions that decide quality on messy enterprise content.
Clarify first (2 minutes). Who asks, through which identity provider? How often do documents change? Who may see what? What is the cost of a wrong answer? Are there scans? Where must data live?
1. Ingestion. Contracts have nested clauses, SOPs have tables and step lists, policies have versions and effective dates. Parse before you chunk: recover headings, section numbers and tables. Scanned files need OCR, and a bad parse caps everything downstream.
2. Chunking. Split on the document's own units (numbered sections, headings, list items) and keep the heading path with the chunk. Typical chunks are a few hundred tokens. Detail is in the chunking card.
3. Metadata. Document ID, version, effective date, section path, source link, and the access list copied from the source system. These enable filtering, citations and debugging.
4. Retrieval. Run BM25 and vector search in parallel and fuse with reciprocal rank fusion. Pull a wide candidate set (for example 50), then rerank down to the 5-10 passages that go in the prompt. Vector search misses exact strings like clause numbers and form codes; BM25 catches them.
5. Generation. Number the passages, tell the model to answer only from them and cite each claim, and to say 'I could not find this' otherwise. Verify citations in code.
6. Measurement. Label real questions with the passages that answer them. Track recall@k on retrieval and faithfulness on answers separately, so you know which layer to fix. Anthropic's contextual retrieval work is a useful reference point: adding chunk context to both embeddings and BM25 cut top-20 retrieval failures by 49% (5.7% to 2.9%), and adding a reranker took it to 67% (to 1.9%) on their test sets.
Scope the first release. One document set, one team, always-cited answers. Say what you are leaving out (multi-language, tables as images, cross-document reasoning).
The line to end on: retrieval quality is mostly decided before the model sees anything, so I would spend the first week on parsing and a labelled question set, not on model choice.
- The first 10 answers in the demo were wrong. What do you look at first?
- How would you handle a 300-page contract where the answer spans two sections?
- What would you cut to ship in two weeks?
- Forward Deployed Engineer Interview Questions and Answers (2026) (prep, 2026-07-15) FDE question: design a RAG system over 40,000 internal documents where quality is contractually tied to the engagement, and where retrieval quality comes from.
- RAG system design interview: a worked FDE answer (prep, 2026-09-25) Worked FDE answer covering ingestion and chunking, hybrid retrieval with reranking, permission filtering, citations, evaluation, cost and latency.
- Forward Deployed Engineer Interview: Prep Guide (prep, 2026-10-24) Decomposition-round version of an internal policy assistant: clarifying questions, components, a first version that always cites, and risks like stale documents.
- Contextual Retrieval in AI Systems - Anthropic (official, 2024-09-19) Contextual embeddings plus contextual BM25 cut top-20 retrieval failures by 49%, and adding reranking by 67%.