Users say the assistant became slow. P95 latency doubled this week. How do you find out why?
Answer it out loud first, then compare. That's what makes it stick.
I break latency into stages using traces: network, retrieval, queue time, time to first token, and total generation. Then I check what changed: input tokens grew, output got longer, a prompt cache stopped hitting, a retry storm began, or the provider is slower. Fixes follow the cause: trim context, cache the stable prefix, stream the response, route easy requests to a smaller model, or cap output length. I verify with before and after percentiles.
- Trace by stage, not one number
- Time to first token vs total time
- Check tokens in and out
- Check retries, queues, cache hits
- Fix the biggest stage, re-measure percentiles
'The model is slow' is a guess. Start with a stage breakdown.
1. Instrument stages
- One trace per request with spans for auth and routing, retrieval, reranking, queue or wait time, the model call, and post-processing. OpenTelemetry's GenAI conventions record operation duration and input and output token counts per call.
- Separate time to first token (TTFT) from total generation time. TTFT grows with input size; total time grows with output length.
2. Compare this week with last week
- Did the median or only the tail move? Tail-only suggests retries, rate limiting or one slow dependency; a median shift suggests more tokens or a slower model.
- Plot input tokens and output tokens over time. Context creep from longer chat history or bigger retrieved chunks is the most common silent cause.
3. Usual suspects
- Prompt growth: more retrieved passages, a longer system prompt.
- Cache misses: prompt caching only helps when the prefix is identical. A timestamp or user name placed early in the prompt breaks it. Anthropic reports caching can cut latency by up to 85% on long prompts, in its own benchmarks.
- Retries: 429s or timeouts retried silently add whole seconds to the tail.
- Output length: a prompt tweak that now asks for explanations.
- Provider: check their status page and compare against another region.
4. Fixes by cause
- Trim and rerank context; move volatile data to the end of the prompt; stream output so perceived latency drops; cap max output tokens; route simple intents to a smaller faster model; parallelise independent tool calls.
5. Verify
- Report p50, p95 and p99 before and after, plus a quality check. Never trade accuracy for speed without running the eval set.
Anthropic's own guidance: get a prompt working first, then optimise, and the main levers are model choice and token count.
- P50 is fine but P99 is terrible. What do you look at?
- How would streaming change what you tell the customer?
- Your fix cut latency 40% but quality dropped. What now?
- Reducing latency (Anthropic docs) (docs, unknown) Defines time to first token and names model choice, token count and streaming as the main latency levers.
- Prompt caching with Claude (official, 2024-08) Reports up to 85% latency reduction on long prompts from caching, in Anthropic's benchmarks.
- Inside the LLM Call: GenAI Observability with OpenTelemetry (docs, 2026) Describes spans, duration and token usage attributes for LLM calls.