A customer wants AI search over their confidential internal documents, and nothing may leave their environment. Design it.
Answer it out loud first, then compare. That's what makes it stick.
I'd start by asking who searches, over what, and what 'good' means. Then I'd draw five boxes inside their network: connectors that ingest from their document stores, a chunk-and-embed pipeline, a vector plus keyword index, an answering service that calls a model they approve, and an audit log. Permissions travel with every chunk so users only see what they may already see. I'd pilot on one document set, measure answer quality on real questions, then widen.
- Clarify users, corpus, and success criteria
- Ingest: connectors, parsing, chunking, embeddings
- Index with ACL metadata (vector + keyword)
- Answer service: retrieve, rerank, cite, call approved model
- Audit, monitoring, eval set
- Pilot on one corpus, then scale
Treat this as a deployment inside someone else's walls, not a greenfield design. You inherit their cloud, identity provider and network rules.
Clarify first
- Who searches (50 analysts or 50,000 staff)? Which systems hold the documents (SharePoint, file shares, a DMS)? How many documents and how often do they change?
- What does a wrong answer cost? Legal research and a canteen menu need different safeguards.
- Where may model inference run: customer VPC, a cloud provider's private endpoint, or the vendor's API under a data agreement?
Architecture
- Connectors read from source systems with a read-only service account, incrementally (change feeds or modified-time cursors), and keep a document ID and source ACL.
- Parsing and chunking: handle PDFs, scans (OCR) and tables explicitly; chunk by structure (headings) with some overlap; store the source link and page.
- Index: embeddings plus keyword (BM25) in one store the customer can operate, such as Postgres with pgvector or OpenSearch. Hybrid search usually beats vectors alone on part numbers and names.
- Answer service: retrieve top 20-50, rerank to 5-8, build a prompt with citations, call the approved model, return answer plus links. If no chunk is relevant, say so.
- Security: ACL filter applied at query time using the caller's identity, encryption at rest with customer keys, private networking to the model endpoint, no document text in general-purpose logs.
- Audit: who asked what, which documents were retrieved.
Prove it
Build a 100-200 question set with the customer's experts (question, expected source document). Measure retrieval hit rate at k and answer correctness before any wide rollout.
Trade-offs to name
Freshness vs cost of re-embedding, strict permissions vs recall, and a self-hosted open model (control, more ops) vs a private cloud endpoint for a frontier model (quality, depends on the provider's retention terms).
End by saying what you would ship in week two: one corpus, one team, thumbs up/down on every answer.
- How do you keep deleted or restricted documents out of answers?
- The corpus is 20 million documents. What changes?
- How would you test retrieval separately from the model?
- The customer wants everything self-hosted. Which model do you pick and why?
- Forward Deployed Engineer System Design Interview: What to Expect and How to Pass (prep, 2026) Lists private-environment document search as a typical FDE system design prompt and names scaling, regulation and monitoring as follow-ups.
- Forward Deployed Engineer Interview Questions (prep, 2026) Gives a customer-firewall deployment example where no data may leave the network.