Before writing any metrics, how do you figure out what to evaluate? Walk me through building a failure taxonomy.
Answer it out loud first, then compare. That's what makes it stick.
I start with error analysis, not a metrics menu. I read around 100 diverse real traces and write a short free-text note on the first thing that went wrong in each. Then I group those notes into five to ten failure categories, count them, and rank by frequency times business cost. The top categories get dedicated evals: code checks where a rule exists, a calibrated judge where judgment is needed. Generic metrics like 'helpfulness' rarely point to a fix.
- Collect ~100 diverse traces
- Open coding: note the first failure
- Axial coding: group into 5-10 categories
- Count and rank by frequency x cost
- Build one targeted eval per top failure
Off-the-shelf metrics (helpfulness, toxicity, BLEU) measure what is easy, not what is broken in this deployment. The method that keeps showing up in practitioner guides is borrowed from qualitative research.
1. Gather traces
- About 100 diverse traces: full input, retrieved context, tool calls, intermediate steps and final output. Sample across intents and channels, plus some flagged by users. If there is no traffic yet, use structured synthetic inputs run through the real system.
2. Open coding
- A domain expert (ideally with you) reads each trace and writes a specific note: 'called the write tool before getting approval', 'gave last year's fee', 'kept troubleshooting after the user asked for a human'.
- Note the first upstream failure. Pipelines are causal; a misread intent early on causes several downstream symptoms. Fix the root and the cascade often disappears.
- Avoid a predefined checklist at this stage; let the data name the problems.
3. Axial coding
- Group notes into 5-10 categories with crisp definitions. An LLM can propose clusters; a human finalises them.
- Count. In one published case (NurtureBoss, via Hamel Husain and The Pragmatic Engineer), three categories, date handling, human-handoff failures and conversation flow, accounted for most issues.
4. Prioritise
- Rank by frequency x severity x fixability. A 2% failure that sends a wrong wire instruction outranks a 15% tone issue.
- Fix the obvious ones immediately (a missing instruction) without building an eval for them.
5. Turn categories into evals
- Rule-checkable failures: code assertions (date in the future, required field present, tool order).
- Judgment failures: one binary judge per category, calibrated on 100-200 labels.
- Each category becomes a tagged slice in the golden set, so you can report 'handoff failures: 11% to 3%'.
6. Repeat until saturation
- Keep reviewing new traces until fresh batches stop revealing new categories, then revisit after any major change. Expect criteria to shift as you read more (the 'criteria drift' finding).
- Who should do the open coding, you or the customer's expert?
- How do you sample traces so you don't only see loud failures?
- How would you use an LLM to speed this up without losing rigour?
- AI Evals: Everything You Need to Know (prep, 2026-09-18) Error analysis via open and axial coding, about 100 diverse traces, 100-200 labels per failure mode for a judge, structured synthetic data, and retrieval metrics like Recall@k and MRR.
- A pragmatic guide to LLM evals for devs (prep, 2025-12-02) Case study where open coding then axial coding showed three failure types (date handling, handoffs, conversation flow) caused most problems.
- An LLM-as-Judge Won't Save The Product - Fixing Your Process Will (prep, 2025-04-20) Evals as the scientific method: look at data, annotate, hypothesize, experiment against a baseline; keep human sampling even with automated judges.
- Who Validates the Validators? (UIST 2024) (docs, 2024-10-11) Identifies criteria drift: people refine their grading criteria while grading outputs, so criteria cannot be fully fixed in advance.