Debugging AI in production
Incidents: hallucination spikes, latency, rate limits, prompt injection, bad outputs, regressions after a model change.
- Overnight, the hallucination rate on our customer's support bot spiked. Walk me through how you debug it. · mediumFree
- Users say the assistant became slow. P95 latency doubled this week. How do you find out why? · mediumFree
- At peak hours your integration starts getting 429 errors from the model provider. What do you do? · mediumFree
- A customer's support agent read a ticket containing hidden instructions and emailed out internal data. How do you respond and prevent it? · hardFree
- After upgrading to a newer model version, the customer's workflow got worse on some tasks. How do you handle it? · hardFree
- An AI agent worked in staging but behaves inconsistently in the customer's production environment. How do you debug it? · mediumPro
- It is 2am and the customer's AI assistant is giving wrong answers to many users. What do you do in the first 30 minutes? · easyPro
- What would you log and monitor for an LLM application so you can debug it later, and what would you avoid logging? · easyPro
- Your pipeline expects JSON from the model, and parse failures jumped from 0.5% to 6%. How do you investigate and fix it? · mediumPro
- The customer's LLM bill tripled this month without a matching rise in usage. How do you find the cause? · mediumPro
- An agent in production sometimes gets stuck calling the same tool over and over. How do you find and stop that? · mediumPro
- A customer's executive sends you one screenshot of a terrible AI answer and says fix it. What do you do? · easyPro
- How do you run a postmortem after an AI incident, and how is it different from a normal software postmortem? · easyPro
- How would you ship prompt or model changes to a customer's live system safely, so a bad change is caught and reversed fast? · hardPro
- The model provider has a partial outage and your customer's product is down. How do you design for this, and what do you do right now? · hardPro