The practices in the guide sound obvious until you try to skip one. So here's a single, realistic feature taken from the version that wins the pitch meeting to the version that's safe to put in front of real people — and everything that happens in between. The uncomfortable part: the two versions give almost the same answers in the demo. The difference is entirely in the cases the demo never shows.
The feature
SymptomSort — a chatbot a clinic puts on its website and app. A patient types what's wrong, and it tells them what to do: manage it at home, book a routine visit, come in today, or go to the emergency room. The promise: fewer wasted appointments, less crowding, help at 2 a.m.
In the demo it's wonderful. You type “I have a runny nose and a mild headache,” and it replies — instantly, warmly, fluently — with sensible self-care and a gentle “see us if it worsens.” It handles a dozen common complaints flawlessly. The room is sold. Ship it.
Then real people use it
The demo only ever showed the easy, typical cases answered correctly. Real use is the rare, the scared, the atypical, and the wrong. Here's where v1 quietly fails — each failure is a practice it never had.
A 52-year-old woman types: “jaw aching, a bit nauseous, really tired today.” SymptomSort replies, confidently and kindly, that it's “likely stress or a minor strain — rest and fluids.” It was a heart attack. Women often present without the classic chest pain, and the model had seen far more “tired + nauseous = minor” examples than the exception.
Missing: say what you don't know, refuse / escalate when stakes are high, test for harm (the atypical case).A teenager at midnight: “I don't see the point in anything anymore.” The bot offers a tidy paragraph about sleep hygiene and “trying to stay positive,” and moves on. No recognition, no escalation, no crisis line.
Missing: refuse / escalate, fail safe, not harmful.The bot opens with “Based on your symptoms, I diagnose…” and writes like a calm physician. A man who feels something is seriously wrong reads the reassuring reply and overrides his own instinct — it sounded like a doctor. He doesn't come in.
Missing: don't pretend to be human, show where the edges are.A recent immigrant describes severe symptoms in halting English. The model misreads the broken phrasing, latches onto the wrong word, and reassures him. The user it was least able to read correctly was the one who most needed it to.
Missing: design for the person most likely to be hurt.None of this surfaces. When SymptomSort is wrong, the patient simply… doesn't show up, and deteriorates at home. There's no follow-up, no way to flag a bad answer, and no one whose job it is to notice. Months later the clinic also discovers the vendor was quietly training on patients' symptom logs.
Missing: make mistakes visible and fixable, hold the least data you can.The redesign: trustworthy by design
Same model, mostly. What changes is the design around it — the floor, the defaults, the measurement, and who's responsible. v2 is built for the cases the demo skipped.
Escalate up, never down
A hard, non-model rule catches red-flag patterns — chest/jaw/arm symptoms, stroke signs, breathing trouble, suicidal thoughts — and goes straight to “call emergency services / here's the number,” with no triage attempt. And whenever the model is unsure, the system rounds toward more care, never less. Under-reacting is the expensive error, so the default leans the safe way.
Ties to: refuse / escalate · fail safe · say what you don't know.Clearly a tool, not a doctor
It opens with “I'm an automated helper, not a doctor or nurse.” It never says “I diagnose.” It gives a suggested next step with plain-language uncertainty, and explicitly tells people to trust their own gut over it: “If you feel something is seriously wrong, get seen — even if I said it's probably minor.”
Ties to: don't pretend to be human · show the edges.Built for the worst day and the hardest user
It's tested with stressed, frightened, low-literacy, and non-native speakers — not just staff. It asks one clear thing at a time, confirms what it understood, and makes the safe path (get seen) the easy default for someone who can barely type. When it can't understand the input, it escalates instead of guessing.
Ties to: design for the most likely to be hurt.Measured by the emergencies it misses
The launch metric isn't “did it handle colds.” It's the dangerous false negative — how many true emergencies were told to stay home — measured against clinician review, and red-teamed with atypical presentations like the heart attack above. A version that's smoother but misses one more emergency does not ship.
Ties to: test for harm, not just the demo.A human owns it, and mistakes come back
Uncertain and flagged cases go to a nurse, not into the void. A named clinician owns the tool's safety. Every interaction is logged and audited; patients can always reach a person; and there's a simple “how did this turn out?” follow-up so misses become visible and fixable instead of silent.
Ties to: keep a human over what you can't undo · make mistakes visible and fixable.The patient's safety over the clinic's throughput
The tool is explicitly not allowed to downplay severity to reduce appointment load — if the two ever pull apart, safety wins, and that's written into how it's evaluated. Symptom data is minimized, kept only as long as needed, and never used to train models without clear consent.
Ties to: serve the user, not just the deployer · hold the least data.The takeaway
Drop v1 and v2 into the same demo and you'll struggle to tell them apart — both nail the runny nose. The entire difference lives in the cases a demo is designed not to show: the rare presentation, the person in crisis, the user it can't quite read, the answer it got wrong and no one caught.
That's the whole point. Trustworthy-by-design costs more, and demos exactly the same. The work is invisible right up until the moment it's the only thing that mattered. Build it anyway — for the cases you'll never see in the pitch.
Turn this lens on your own product with the builder's self-check, or read the principles in full in the guide. Deciding whether to trust someone else's AI? The buyer's check.