Debugging AI in productionPractice question (not reported)mediumChecked 2026-11-17
At peak hours your integration starts getting 429 errors from the model provider. What do you do?
Answer it out loud first, then compare. That's what makes it stick.
30-second answer
First I read the error: which limit was hit, requests per minute or tokens per minute, and is there a retry-after header? Short term I add exponential backoff with jitter that respects that header, and queue work so bursts are smoothed. Then I reduce token use, shift non-urgent jobs to batch, and ask the provider for a higher tier. I also check whether it is a spend cap rather than a rate limit, because retrying will not help there.
Framework to remember
- Read the 429: which limit, retry-after
- Backoff with jitter, honour retry-after
- Queue and smooth bursts
- Cut tokens; batch non-urgent work
- Raise limits; add fallback
Deep answer
A 429 is information, not just an error.
1. Diagnose
- Anthropic's docs say exceeding any limit returns a 429 describing which limit was exceeded, with a
retry-afterheader giving the wait. Limits are measured in requests per minute, input tokens per minute and output tokens per minute, per model class. - Find which dimension you hit. Few huge requests hit tokens; many tiny ones hit requests.
- Rule out a spend or quota cap: those can also come back as 429 yet retrying will not help. Check the message.
- Limits apply at the organisation level, so another team's batch job can starve your production traffic.
2. Stop the bleeding
- Exponential backoff with jitter, using retry-after as the minimum wait. Retrying instantly from many workers causes a thundering herd.
- Cap retries and total wait; fail gracefully with a message the user can act on.
3. Smooth the demand
- Put a queue and token-bucket limiter in front of the model client so you stay under the limit proactively instead of reacting to errors.
- Prioritise: interactive chat first, back-office summarisation later.
4. Reduce consumption
- Trim prompts, cache stable prefixes (cached reads are often not counted toward input token limits on some providers; check current docs), lower max output tokens, and route simple tasks to a smaller model with a separate limit.
- Move overnight and bulk workloads to a batch API.
5. Capacity
- Request a tier increase with real numbers: peak RPM, tokens per request, growth forecast. Consider a second provider or cloud-hosted route as a fallback for the same model family.
6. Observe
- Dashboard 429 rate, retry count, and queue depth. Alert before 429s hit users.
Customer framing: explain that this is a capacity plan, and show the peak numbers behind the request.
Strong answer
Identifies which limit and whether it is a spend cap, adds jittered backoff, proactive queueing and token reduction, and plans capacity with numbers.Weak answer
Wraps the call in a naive retry loop with no jitter, which amplifies the problem.The trap
Retrying instantly in a tight loop, or retrying a spend-cap 429 that can never succeed.Likely follow-ups
- Your retries are making the outage worse. Why, and how do you stop it?
- Which traffic do you shed first when you cannot serve everything?
- How do you size a limit increase request?
Sources (2) · confidence medium
- Rate limits (Anthropic docs) (docs, unknown) States that exceeding a limit returns 429 with a retry-after header, and limits are in RPM, input and output tokens per minute.
- AI API 429, 529 and 503 Errors: Retry Rules That Match the Docs (prep, 2026) Retry guidance for 429, 529 and 503 errors across providers.