Evals & measuring LLM quality
Designing eval sets, LLM-as-judge and its calibration, offline vs online metrics, regression testing, golden sets, failure taxonomies.
- You've built an AI feature for a customer. How do you know it's actually working? · easy · reported 2×Free
- The customer has no labeled data and no eval set. How do you build one in the first two weeks? · medium · reported 3×Free
- Can you use an LLM to grade your outputs? How do you make sure the judge is trustworthy? · medium · reported 3×Free
- What biases do LLM judges have, and how would you detect and control each one? · hardFree
- You changed the system prompt and the customer asks, 'Did it get better?' Walk me through how you answer. · medium · reported 2×Free
- Your eval score went from 82% to 85% on 300 examples. Is that a real improvement? · hardFree
- What's the difference between offline and online evaluation, and what would you monitor after launch? · easy · reported 1×Free
- What is a golden set, and how do you keep it useful over the life of a deployment? · medium · reported 1×Free
- Before writing any metrics, how do you figure out what to evaluate? Walk me through building a failure taxonomy. · mediumFree
- A key quality metric for our customer-facing AI agent dropped last week. How do you diagnose it and decide what to fix first? · hard · reported 2×Free
- How do you evaluate a multi-step agent that calls tools, not just a single LLM response? · hardFree
- How would you evaluate a RAG system before launch? · medium · reported 1×Free
- Exact match, code checks, LLM judges, human review: how do you decide which grader to use for what? · easyFree
- How do you agree launch criteria with a customer? What's your release threshold? · easy · reported 2×Free
- Your eval scores keep going up, but the customer says the product isn't getting better. Which do you trust, and what do you do? · medium · reported 1×Free