A customer says 'we want an agent for this.' How do you decide whether it needs an agent loop at all?
Answer it out loud first, then compare. That's what makes it stick.
I start by asking what decision the output feeds and what a wrong answer costs. If the steps are known in advance, I build a workflow: fixed code paths with an LLM call at each step, maybe a router. I only use an agent loop when the number and order of steps genuinely cannot be predicted. Loops cost more, are slower and are harder to test, so I earn the autonomy: start with the simplest thing that works, then add a loop where measurements show it helps.
- Ask: what decision, what cost of error, where is ground truth
- Known steps: workflow (chain, route, parallelize)
- Unknown steps: agent loop with tools and stop rules
- Compare cost, latency and testability
- Start simple, add autonomy only where evals justify it
The first move is a classification, not a build. Anthropic separates two things: workflows, where LLMs and tools follow predefined code paths, and agents, where the model directs its own process and tool use. Most customer problems are workflows wearing an agent costume.
Questions that decide it
- Can I write the steps on a whiteboard before seeing the input? If yes, use a prompt chain with programmatic gates between steps.
- Are there a few distinct input types needing different handling? Use routing, for example easy tickets to a small model and hard ones to a larger one.
- Is the path genuinely open-ended (debugging an unfamiliar system, research across unknown sources)? Only then is a loop justified.
What the loop costs you
- Every extra step resends history, so cost grows faster than step count.
- Latency is the sum of sequential model calls; 8 steps at 3-5 seconds each is a 30-40 second wait.
- Testing is harder because two runs of the same task take different paths.
- Errors compound: at 95% per-step reliability, ten steps succeed about 60% of the time.
What the loop buys you Flexibility on tasks you cannot enumerate, and the model gets 'ground truth' from tool results at each step. Good agents also pause for human feedback at checkpoints and have stopping conditions such as a maximum number of iterations.
What I would say to the customer 'We will ship a workflow first because you can audit it, then give the model freedom only at the step where the rules run out.' This also protects against framework temptation: Anthropic advises starting with direct API calls, because frameworks hide the prompts and make debugging harder.
Evidence to bring. A classifier plus a template often beats a loop on cost and accuracy; measure both on the customer's own cases before upgrading.
- Give an example where you would convert an agent back into a workflow.
- Where in a workflow would you still let the model choose the next step?
- How would you show the customer the loop is worth its extra cost?
- Building effective AI agents - Anthropic (official, 2024-12-19) Defines workflows vs agents, recommends simplicity, direct API use over heavy frameworks, and stopping conditions and human checkpoints for agents.
- Real Interview Questions for the Forward Deployed Engineer at OpenAI (prep, 2026-11-15) Frames the OpenAI FDE design fork as single call vs agent loop, and notes a classifier plus template often beats a loop.