A hospital customer says no patient data may leave their network. How do you decide where the model runs?
Answer it out loud first, then compare. That's what makes it stick.
I list the real options: an open-weight model self-hosted on their GPUs, a frontier model through their cloud provider's private endpoint, or a vendor API under a data agreement. I ask what 'leave the network' really means to their compliance team, because private endpoints and retention terms differ. Then I compare quality on their own test cases, cost, and operations burden. If quality demands a frontier model, I push to get a private-cloud route approved rather than defaulting to weaker self-hosting.
- Pin down what the constraint literally means
- List options: self-hosted, private cloud endpoint, vendor API
- Check retention, residency and contract terms per option
- Benchmark on the customer's own tasks
- Weigh quality, cost, ops burden; recommend and stage
The trap is treating 'on-prem or nothing' as given. First find out which rule sits behind the sentence: a regulation, a contract, or a security team's preference. They allow different solutions.
Options
- Self-hosted open-weight model on customer GPUs. Data never leaves, no per-token fee, but you own serving (vLLM-style inference servers, GPU capacity, upgrades), and quality on hard tasks may trail frontier models. A 70B-class model needs several high-memory GPUs, so ask what hardware exists before promising it.
- Frontier model through the customer's cloud provider (Bedrock, Vertex or a similar managed service) reached over a private endpoint inside their account. Traffic stays on the provider's network and their existing identity and key management apply.
- Vendor API directly, under a contract. Anthropic's docs say zero data retention is enabled per organization after review, and that it covers the direct platform, while cloud-partner deployments follow that partner's own retention policy. So the answer depends on which route you pick; read the current terms, do not assume.
How I decide
- Compliance fit: does the legal team accept a business associate agreement or equivalent for that route? Retention windows, logging defaults and region all matter.
- Quality: run 100+ of their real tasks (de-identified) on each candidate. A model that is 10 points worse on their eval may cost more in clinician time than it saves in infrastructure.
- Cost: GPU hardware and an on-call burden versus per-token fees. Do the arithmetic at expected volume, not at pilot volume.
- Operations: who patches the inference server and swaps models in a year?
Recommendation shape
Start with the route that can be approved fastest and still pass the eval, design the app behind a thin model-gateway interface so the model can be swapped, and keep de-identification in front of any external call as defense in depth.
Say out loud that you would verify retention and logging terms with the provider's current documentation before committing.
- They have no GPUs and a 6-month procurement cycle. What now?
- How do you de-identify text before it reaches a model, and how do you check it worked?
- The model provider changes its retention policy next year. What protects you?
- Forward Deployed Engineer Interview Questions (prep, 2026) Gives a healthcare example: run a model behind the customer's firewall with no data leaving the network.
- Zero data retention (Claude Code docs) (docs, 2026) States ZDR applies to Anthropic's direct platform and that Bedrock, Google Cloud and Foundry deployments follow those platforms' retention policies.
- API and data retention (docs, 2026) Lists partner-operated platforms as outside direct-platform retention guarantees.