How would you evaluate a RAG system before launch?
Answer it out loud first, then compare. That's what makes it stick.
I evaluate retrieval and generation separately, because they fail for different reasons and need different fixes. For retrieval I build question-to-relevant-document pairs and measure recall at k and MRR. For generation I check faithfulness, meaning every claim is supported by the retrieved context, answer correctness against expert answers, citation accuracy, and whether it refuses when the answer isn't in the documents. Then an end-to-end set from real user questions, sliced by document type.
- Split retrieval from generation
- Retrieval: recall@k, MRR on labelled pairs
- Generation: faithfulness, correctness, citations
- Refusal on unanswerable questions
- End-to-end set from real questions, sliced
The key move is decomposition: a wrong answer is either a retrieval miss or a generation mistake, and you cannot fix what you cannot locate.
Retrieval evaluation (a search problem)
- Build a set of queries paired with the documents or chunks that answer them. Sources: real user questions with expert-marked passages, or reverse-generated questions from chunks (pick a passage, extract a fact, generate the question), then review a sample.
- Metrics: Recall@k (is the right chunk in the top k?), Precision@k, MRR (how high is the first relevant chunk?). Recall@k matters most because the generator cannot use what it never sees.
- Slice by document type (PDF tables, scanned forms, emails) and by query type (lookup, comparison, multi-hop). Tables and scans usually lag.
Generation evaluation
- Faithfulness / groundedness: share of answer claims supported by the retrieved context. Ragas defines faithfulness exactly this way.
- Answer correctness: against expert reference answers, by code for exact facts (dates, amounts) and a calibrated judge for prose.
- Citation accuracy: does each cited passage exist and support the sentence it is attached to? Mostly code-checkable.
- Context utilisation / recall: did the answer use the key facts that were retrieved?
Negative cases
- Questions whose answer is not in the corpus. The right behaviour is to say so. Measure false-answer rate here; it is where most hallucinations live.
- Permission tests: a user without access should not get content from restricted documents.
End to end
- 200-400 real questions, stratified, with expert answers. Report per slice.
- Track latency and cost per answer, since reranking and larger k cost money.
Decision rules
- Low recall@k: fix chunking, hybrid search, reranking, metadata filters.
- High recall but low faithfulness: fix the prompt, context ordering, or model.
- Good on both but users unhappy: wrong questions in your set; go back to traffic.
- Recall@5 is 95% but faithfulness is 70%. Where do you look?
- How do you label relevant chunks when chunking keeps changing?
- How do you evaluate freshness when documents are updated daily?
- Forward Deployed Engineer Interview Questions (2026) (prep, 2026-10-22) FDE question list including evaluating a RAG system before launch, defining success metrics, and monitoring an LLM app in production.
- AI Evals: Everything You Need to Know (prep, 2026-09-18) Error analysis via open and axial coding, about 100 diverse traces, 100-200 labels per failure mode for a judge, structured synthetic data, and retrieval metrics like Recall@k and MRR.
- Faithfulness - Ragas docs (docs, 2026-11-13) Defines faithfulness as the share of response claims supported by the retrieved context.
- Context Recall - Ragas docs (docs, 2025-12-09) Defines context recall as how many reference claims or documents the retriever actually surfaced.