How do you agree launch criteria with a customer? What's your release threshold?
Answer it out loud first, then compare. That's what makes it stick.
I set thresholds with the customer before building, tied to the cost of a mistake and today's baseline, not to a round number. First I measure how their current process performs, human or rules. Then we agree per-criterion thresholds: zero tolerance for things like PII leaks, a target for task accuracy that beats the baseline with a confidence margin, plus latency and cost limits. Below threshold, a slice routes to humans rather than blocking the whole launch.
- Measure today's baseline
- Price the cost of each error type
- Per-criterion thresholds: guardrails vs targets
- Include CI margin, latency, cost
- Staged rollout with human fallback per slice
Launch criteria are a negotiation grounded in numbers the customer already believes. OpenAI's FDE job descriptions describe exactly this work: defining launch criteria against customer-specific acceptance thresholds.
1. Baseline first
- Measure the current process on the same golden set: human agents, the old rules engine, or nothing. Humans are rarely 100%; expert reviewers often disagree on 5-15% of nuanced cases. 'Better than today with evidence' beats 'perfect'.
2. Price the errors
- Not all errors cost the same. A misrouted ticket costs minutes; a wrong medication dose or a wrong wire is a different category.
- Separate false positives and false negatives; a fraud flagger and a refund approver want very different trade-offs.
3. Two kinds of threshold
- Guardrails (ship-blockers): zero tolerance or near it. Example in the style of Anthropic's docs: under 0.1% of outputs flagged for a safety issue across 10,000 trials. PII leakage, missing required disclosures, actions taken without approval.
- Targets: task accuracy 90% or more on the holdout, with the lower bound of the 95% interval above the baseline; override rate under an agreed level in pilot.
- Operational: p95 latency, cost per task, availability.
4. Make it measurable before building
- Write each criterion as: metric, dataset, grader, threshold, owner. Sign it off with the customer's business owner and risk/compliance.
- Anthropic's docs list good criteria as specific, measurable, achievable and relevant; vague terms like 'accurate' do not survive a launch review.
5. Partial launches
- If 80% of intents meet the bar and 20% don't, launch the 80% and route the rest to humans. Per-slice thresholds turn 'not ready' into 'ready for these cases'.
- Stage it: shadow mode, then a small percentage of traffic or one team, with rollback triggers defined in advance.
6. After launch
- The same thresholds become production alerts. Re-validate on every model, prompt or retrieval change.
- The customer insists on 99% accuracy. How do you respond?
- Who on the customer side should sign off on the criteria?
- How do thresholds change when the provider updates the model?
- FDE Interview Questions and Enterprise AI Scenarios (prep, 2026-08-05) FDE question list including how to build a golden dataset, whether an LLM can judge another LLM, and what your release threshold is.
- Forward Deployed Engineer Interview Questions (2026) (prep, 2026-10-22) FDE question list including evaluating a RAG system before launch, defining success metrics, and monitoring an LLM app in production.
- Forward Deployed Engineer (FDE), Healthcare - SF (OpenAI careers) (official, unknown) Role description: define evaluations, human-review workflows and launch criteria against customer-specific acceptance thresholds.
- Define success criteria and build evaluations - Claude Platform Docs (docs, unknown) Official guidance: specific, measurable success criteria; task-specific evals with edge cases; prefer code grading, then LLM grading tested for reliability, human grading last.