Build a small pipeline: read customer documents, call an LLM to extract fields, and handle the case where the model returns malformed output.
Answer it out loud first, then compare. That's what makes it stick.
I define the exact schema I want first, then ask the model for structured output and validate the reply in code. If it fails validation I retry once with the error message fed back, and if it still fails I route the document to a review queue instead of guessing. I keep each step small: load, extract, validate, store. I log the input id, model, prompt version and outcome, so I can measure the failure rate and fix the worst cases.
- Define the target schema
- Extract with structured output
- Validate in code, never trust the model
- One repair retry, then review queue
- Log ids, prompt version, outcomes
This is the shape of the live exercise Distyl-style loops describe: messy input, an LLM call, a malformed reply, a working pipeline. The bar is 'it runs and handles failure', not cleverness.
Pipeline stages:
- Load documents and skip unreadable ones with a reason.
- Extract with a prompt that includes the schema and 1-2 examples, temperature 0.
- Validate the output with a schema library such as Pydantic. Check types, required fields, enums and value ranges, not just that it is JSON.
- Repair or route. One retry that includes the validation error. After that, queue for human review.
- Store results with the source id and a prompt version.
import json
from pydantic import BaseModel, ValidationError
class Invoice(BaseModel):
vendor: str
total: float
currency: str
def extract(doc, llm, retries=1):
prompt = f"Return JSON with vendor,total,currency.\n{doc}"
for _ in range(retries + 1):
raw = llm(prompt)
try:
return Invoice(**json.loads(raw)), None
except (json.JSONDecodeError, ValidationError) as e:
prompt += f"\nYour last reply was invalid: {e}. Fix it."
return None, "needs_review"
Malformed output happens in several ways: prose around the JSON, trailing commas, missing keys, a number as a string, or a confident hallucinated value. Native structured-output or tool-calling features cut the first kinds sharply but do not remove the last; always validate meaning too (does the total equal the sum of line items?).
Operational points:
- Cost: a 1,000-document run at roughly 2,000 tokens each is about 2M input tokens; state the estimate and use a cheaper model for first pass if accuracy holds.
- Concurrency with a cap, plus the retry logic for 429s.
- A gold sample of 30-50 hand-checked documents to measure field-level accuracy before scaling.
- Never let the model's output become a SQL string or a shell command.
Close by reporting rates: parsed first try, parsed after repair, sent to review.
- The JSON is valid but the total is wrong. How do you catch that?
- How would you run 100,000 documents cheaply?
- How do you decide the prompt change did not make things worse?
- How Distyl AI hires forward deployed engineers (prep, unknown) Describes parsing messy input, calling an LLM API, handling malformed output and assembling a working pipeline in the coding round.
- Python skills for the forward deployed engineer interview (prep, 2026-09-18) Lists a small RAG pipeline (chunking, embedding, retrieval, passing context to a model) among FDE Python tasks.