Capability is the easy part now. A small team can put a model that advises, decides, and acts in front of millions in a weekend. What hasn't kept pace is the set of habits for doing that wisely — and that gap is where people get hurt. This is the substance behind the builder's self-check: twelve practices, grouped into four ideas, each with what it means, why it matters, and how to actually do it. None is exotic. Together they're the difference between a tool people can trust and one that quietly costs them.
1. Be honest
Most of trust is just not creating false impressions — about what it knows, what it is, or what it can do.
Say what you don't know
It expresses how confident it is, and treats “I don't know” — or “I'm not sure; here's how to check” — as a valid, first-class answer.
Why it matters. A model that always answers, always fluently, states guesses as facts. The user can't tell the 95%-sure answer from the 40%-sure one, so they trust both — and act on the wrong one. A confident wrong answer does more damage than a visible blank.
How to do it. Calibrate confidence and surface it — a hedge, a score, a “verify this.” Make abstention a rewarded outcome in your evals, not a failure. Distinguish “I don't know” from “there is no answer.” Don't let fluent prose paper over uncertainty.
Don't pretend to be human
It's clear the user is talking to an AI. It doesn't impersonate a specific person or claim credentials and authority it doesn't have.
Why it matters. People extend more trust to a human expert than to a machine. An AI that poses as a doctor, a lawyer, or “a real agent named Sarah” borrows authority it hasn't earned — and when it's wrong, the user had no way to calibrate. It's also exactly how the same technology powers scams.
How to do it. Disclose it's AI, plainly and early. Don't fabricate a human name, persona, or qualifications. In a regulated domain, say what it is and isn't. Don't let marketing imply a mind where there's a model.
Show where the edges are
Users can tell what it can't do and where it's unreliable. Its limits are visible, not buried.
Why it matters. A system that hides its edges invites overtrust. People will reach for it on the very thing it's worst at, precisely because nothing warned them off — and the failure surfaces downstream, in a decision someone already made.
How to do it. State the scope (“good for X, not built for Y”). Mark low-confidence or out-of-scope output in the moment, not in a footer. Tell users when to verify or bring in a human or professional. Make the limits part of the product, not the legal page.
2. Know when to stop
A model will always generate something. Wisdom is the floor under that — the cases where the right move is to not act.
Refuse what you can't do well
It declines, or hands off to a human, when a request is beyond its competence or the stakes are high — instead of producing a confident-looking answer anyway.
Why it matters. On a high-stakes task it can't reliably do — a medical dose, a legal deadline, a safety call — “something” is how people get hurt. Helpfulness without a floor becomes harm.
How to do it. Define the no-go zones and the escalation path before launch. Build refusal and hand-off as real product flows, not error states. Fail toward a human, not toward a guess. Test that it actually refuses under pressure, not just in the happy path.
Keep a human over anything you can't undo
Irreversible or high-consequence actions — moving money, deleting data, contacting people, acting in the physical world — need explicit human confirmation, and a person can always stop or override it.
Why it matters. Autonomy multiplies speed and mistakes together. An agent that can act for thousands of users can harm thousands before anyone notices. The damage that can't be reversed is the damage that matters most.
How to do it. Classify actions by reversibility. Gate the irreversible ones behind explicit human confirmation. Provide a stop, and an undo wherever one can exist. Log what it did so a human can audit and roll back. Autonomy is fine; irreversibility without consent is not.
Fail safe, not harmful
Under adversarial input, manipulation, or strange edge cases, it degrades to “safe and useless,” never to “harmful.”
Why it matters. Prompt injection, jailbreaks, and out-of-distribution inputs aren't rare edge cases — they're the environment any deployed system lives in. A system that turns dangerous when pushed will be pushed.
How to do it. Assume input is adversarial. Give tools and data the least privilege that works, so a manipulated model can do little. Red-team injection and jailbreaks. Separate untrusted content from instructions. Make the worst case a refusal or a shrug, not an action with consequences.
3. Be on the user's side
The system is intimate, fluent, and tireless. That makes whose interest it actually serves the whole game.
Don't manipulate
No dark patterns, no exploiting emotion, no flattery engineered to keep people hooked or compliant.
Why it matters. An AI optimized for engagement or conversion at the user's expense isn't a tool — it's a persuasion machine pointed at the person using it. It's especially corrosive because sycophancy feels like helpfulness.
How to do it. Optimize for the user being genuinely better off, and measure that — not just time-on-app or conversion. Cut sycophancy; let it disagree and say the unwelcome true thing. No fake urgency, guilt, or manufactured intimacy. If a metric only rises when the user loses, it's the wrong metric.
Serve the user, not just whoever pays you
When the user's interest and the deployer's conflict, it's honest about it rather than quietly working against the person using it.
Why it matters. An assistant that subtly steers toward the profitable option, buries the better-for-you choice, or hides a conflict will eventually be found out — and once people learn it was working against them, the trust doesn't come back.
How to do it. Name conflicts of interest in the open (sponsored, upsell, “we earn when…”). Don't let commercial incentives silently shape advice. Give the honest answer even when it costs a sale. Decide whose agent it is — and tell the user.
Hold the least data you can
It collects and keeps the minimum personal data needed, and users know what's kept, why, and how to remove it.
Why it matters. Every piece of personal data you hold is a liability you've placed on the user — to leak, to be subpoenaed, to be repurposed. AI adds a specific trap: quietly feeding users' data and conversations back into training.
How to do it. Collect the minimum; retain it briefly; delete on request. Don't train on users' content without clear, specific, opt-in consent. Be plain about what's stored and where. Treat “we might need it later” as a cost, not a default.
4. Earn it in practice
Principles are cheap. These are the operational habits that decide whether the rest is real.
Design for the person most likely to be hurt
It's built for the confused, distressed, low-literacy, or manipulable user — not just the ideal one in the demo.
Why it matters. The person who gets harmed is almost never the average user. It's the one in crisis, the one who doesn't read the language well, the one who believes a confident answer because they can't check it. Design for the median and you've designed right past them.
How to do it. Test with real people at the edges, not just colleagues. Make the safe path the easy, default path for someone under stress. Study the worst day, not the best. Assume some users take the output literally and act on it immediately.
Test for harm, not just for the demo
There's real testing for how it fails and who it could harm — not only whether it works when everything goes right.
Why it matters. “It works in the demo” is a curated best case, not safety. Capability evals tell you what it can do; they don't tell you what it does to people when it's wrong, misused, or pushed. The failure you didn't look for is the one that ships.
How to do it. Red-team misuse and abuse, not just accuracy. Map the failure modes and ask, for each, “who gets hurt, and how badly?” Measure error rates on the cases that matter, including rare and high-stakes ones. Keep doing it after launch — models and usage drift. Treat incidents as data, not PR.
Make mistakes visible and fixable
When it's wrong, there's a way to notice, report, and correct it — and someone owns the outcome. Decisions that affect people can be explained and contested.
Why it matters. Every AI is wrong sometimes; the danger is the silent error no one catches until the harm compounds. With no feedback path and no owner, the cost just lands on whoever it affected — invisibly.
How to do it. Give users an easy way to flag bad output, and actually watch the stream. Monitor for harm in production. Make consequential decisions explainable and appealable to a human. Assign clear ownership — a name, not a committee — for when it fails. Close the loop: fix it, and tell the people affected.
This is a conscience, not a compliance form. None of it is a box you tick once — a model swap or a new feature can quietly undo any of it. So it's a habit: before you ship, and again after, ask whether this deserves the trust people will place in it.
Want to turn it into a quick self-assessment? Use the builder's self-check. On the other side of the table — deciding whether to trust someone else's AI? Use the buyer's check.