How do you chunk real enterprise content: contracts, a wiki page with a long pricing table, and support-ticket threads?
Answer it out loud first, then compare. That's what makes it stick.
I parse first, then split on each document's own structure instead of a fixed token count. Contracts split at numbered clauses with the heading path attached. Tables keep their header row with every group of rows. Tickets split per message, with the title and status as metadata. I cap chunks at a few hundred tokens with small overlap, prepend a short context line, and keep a link to the parent section so I can pull in neighbours when a chunk is too thin.
- Parse first: headings, tables, section numbers
- Split on structure, cap size, small overlap
- Carry heading path and table headers into the chunk
- Add short chunk context before embedding and BM25
- Test chunk settings on the labelled question set
'I would chunk at 512 tokens with overlap' is a default, not a design. Show that chunking depends on the document type and the questions asked.
Step 0: parse well. No chunk size fixes a bad parse. Recover headings, section numbers, list structure and tables; OCR scans. Check a sample by eye before you tune anything.
By content type
- Contracts and policies. Split at numbered clauses or sections. Keep the heading path, for example Exclusions > 4.2 Flood, on each chunk so an exclusion never floats free of its section. A short memo can stay as one chunk.
- Tables. Split by rows but repeat the header row in every chunk, so rows 40-50 still say what the columns mean. For very wide tables consider rendering each row as a sentence ('Enterprise tier: 500 seats, 99.9% SLA').
- Ticket threads. One chunk per message, with ticket title, status and product as metadata. A resolution message is often the useful chunk.
- Slides. One slide per chunk with the deck title.
- Code or config. Split by function or block.
Size. A few hundred tokens with a small overlap is a sound starting range. Too large dilutes the match and wastes context; too small loses meaning. For reference, OpenAI's hosted file search defaults to 800-token chunks with 400 overlap and lets you set 100-4,096, which tells you the range vendors consider reasonable, not what is right for your corpus.
Give every chunk its context. Chunks often do not name their own topic ('the limit is 5,000'). Two cheap fixes: prepend the section path to the text before embedding, or generate a one or two sentence context for each chunk with an LLM. Anthropic reports this contextual approach cut failed top-20 retrievals by 35% with embeddings alone and 49% when also used for BM25. Prompt caching makes the one-off generation cost practical for large corpora.
Parent-child. Retrieve small chunks for precision, then expand to the parent section or neighbouring chunks when the generator needs more room.
Decide with data. Build 100 real questions with their answer passages and compare two or three chunking schemes on recall@k. The best scheme is the one that wins on the customer's documents, not on a blog's benchmark.
- Your table chunks retrieve fine but the answer picks the wrong row. What next?
- What does contextual retrieval cost to run on 10 million chunks?
- When would you use semantic chunking over structure-based splitting?
- OpenAI Interview: Design an Enterprise RAG Assistant | The Forward Deployed (prep, 2026-09-28) Deep dive on chunking enterprise content: split by structure, repeat table headers, one chunk per ticket message, prepend section path.
- RAG system design interview: a worked FDE answer (prep, 2026-09-25) Parse before chunking, split on the document's own units, carry the heading path, and 'a default, not a design' on fixed 512-token chunking.
- Forward Deployed Engineer Interview Questions and Answers (2026) (prep, 2026-07-15) FDE answer stating ingestion and chunking is where most quality is won or lost on contracts, SOPs and policies.
- Contextual Retrieval in AI Systems - Anthropic (official, 2024-09-19) Contextual embeddings cut top-20 failure rate by 35%, with contextual BM25 by 49%; chunk boundaries and overlap affect retrieval.
- Retrieval | OpenAI API (docs, unknown) Defaults of 800-token chunks with 400 overlap, configurable from 100 to 4,096 tokens.