Explain RAG to the customer's head of engineering. When would you pick it over fine-tuning for their private data?
Answer it out loud first, then compare. That's what makes it stick.
RAG gives the model a read path into the customer's data at question time: I index their documents, retrieve the few passages that matter for each query, and the model answers from those, with citations. I pick it over fine-tuning when the goal is knowledge access, because documents change daily, permissions differ per user, and answers must be traceable. Fine-tuning teaches behaviour, like format or tone, not facts. My default is a strong base model plus RAG plus good prompting.
- Model = reasoning engine; RAG = read path to data
- Freshness: re-index in minutes vs retrain
- Access control: filter per user vs weights leak to all
- Attribution: citations make answers debuggable
- Fine-tune only for behaviour gaps evals prove
This is the warm-up question, and the point is to show you can explain an architecture choice to a technical buyer in operational terms, not buzzwords.
The one-sentence model. The LLM is a stateless reasoning engine. RAG adds a read path to the customer's data: at query time you search an index, put the best passages in the prompt, and ask the model to answer only from them.
Why RAG wins for knowledge questions (policy Q&A, support over product docs, contract lookup):
- Freshness. When a policy changes you re-index that document in minutes. A fine-tuned model stays stale until the next training run.
- Access control. Retrieval can filter by the asking user's permissions. Facts baked into weights are available to everyone who can call the model, which is usually disqualifying in an enterprise.
- Attribution. Each answer can point at a passage, so users can verify and you can debug a wrong answer by looking at what was retrieved.
- Iteration speed and cost. Changing chunking or the prompt takes hours and a re-run of your eval set. A training run takes data prep, compute and a re-validation cycle.
Where fine-tuning earns its place:
- Teaching behaviour: a consistent voice, a strict output format, a classification skill.
- Distillation to a smaller, cheaper model at high volume once a prompt-plus-RAG system already works.
- A gap your evals show prompting cannot close.
They compose. A common mature setup is RAG for facts plus a lightly tuned or well-prompted model for behaviour.
When RAG is the wrong tool. If the data fits comfortably in the context window and rarely changes, just put it in the prompt (with prompt caching). If the question needs aggregation over structured data ('total refunds last quarter'), use SQL or a tool call, not similarity search.
Tell the head of engineering the real trade-off: RAG moves the risk from the model to the pipeline. You now own ingestion, permissions, freshness and retrieval quality, and each one needs a metric.
Close with a decision rule: knowledge that changes, differs by user, or needs citations means RAG; behaviour or style the model keeps getting wrong means fine-tune after measuring.
- The whole corpus is 200 pages and changes monthly. Do you still build a vector index?
- When would you combine RAG and fine-tuning in one system?
- What new failure modes does RAG introduce that a plain prompt did not have?
- Forward Deployed Engineer Interview Questions and Answers (2026) (prep, 2026-07-15) FDE question bank that asks to explain RAG to a head of engineering and when to choose it over fine-tuning, citing freshness, access control and attribution.
- Forward Deployed Engineer Interview Questions (2026) | ADaSci (prep, 2026-10-22) FDE question list including what RAG is in a customer project and when to fine-tune instead of using RAG or prompting.
- Retrieval | OpenAI API (docs, unknown) Hosted file search over vector stores with semantic plus keyword search, a vendor-documented RAG read path.