Calibrated Reliance: Stop AI Hallucinations with Better UX

A seamless review screen is why your team waves hallucinated agent drafts through. Design for calibrated reliance: friction where a wrong draft costs the most.

If your sales, CS or ops staff check what an agent wrote before it goes out, you are likely under pressure to make that check "frictionless": an instant draft, a clean screen, one click to approve. That "seamless" experience is precisely why your reviewers are falling for hallucinations.

In the era of deterministic software, friction was a bug. In the era of probabilistic AI, risk-weighted friction is a safety feature.

When your interface looks like an Oracle, authoritative, confident, and instant, your reviewers will treat it like one. When the model inevitably hallucinates (and it will), a "seamless" UI effectively launders that error, presenting a stochastic guess as a verifiable fact. This isn't just a model failure; it is an interaction failure. It creates automation bias and destroys trust when it matters most.

Strategic Insight: Appropriate reliance is a system property that emerges from UX, incentives, measurement, and operational policy, not from model quality alone. A CHI 2025 study by Bo, Wan & Anderson found that while UI interventions reduce over-reliance, they generally fail to improve appropriate reliance.

Key takeaways

Calibrated reliance means your team trusts the agent's draft when it's right and catches it when it's wrong.
Match friction to risk: light signals for internal drafts, mandatory checks for anything costly or hard to undo.
The minimum viable review screen needs an abstention state, micro-checks, a safe halt and a retry limit.

What is calibrated reliance, and why does it matter to reviewers?

When a database returns a row, it is factual. Reviewers bring that same mental model to an agent's draft. Research from Microsoft and Harvard suggests that the goal of AI UX is not "maximum trust," but "appropriate reliance." Your review screen must do three things: help reviewers understand what the agent can and cannot do, signal when to verify, and reduce the "interaction cost" of checking facts so reviewers actually do it.

Risk is not just about the task; it is about the reviewer. Over-reliance risk rises for new starters who lack the domain expertise to spot subtle errors, and for anyone using the agent to do something they don't know how to do themselves. A "Low Stakes" summary becomes "High Risk" if the reviewer cannot check the source material.

Warning: the risk of "placebo transparency"

Showing citations or confidence scores isn't enough. Microsoft Research found that explanations increase reliance on both correct and incorrect answers: a fluent rationale reads as proof, whether or not the answer is right. Crucially, verification aids themselves can hallucinate. A citation nobody has checked only looks like evidence.

How do you decide how much friction a draft needs?

The NIST Generative AI Profile (NIST AI 600-1) lists confabulation and human-AI configuration, which covers automation bias and over-reliance, among the 12 risks of generative AI. Friction is not universally good. If you add "Click to Verify" to a brainstorming tool, you kill utility. But if you strip friction from a contract reply or a refund decision, you invite a costly mistake.

To operationalise this, use a derived risk score to set the controls on the review screen:

Risk Score = (Stake Impact) × (Reversibility Cost) × (User Vulnerability + Automation Level) / Error Detectability

Rare or subtle errors are harder to detect, so they need higher friction.

Risk level	Context	Reversibility	UI strategy	Required friction
High stakes	Contract terms, pricing, refunds	Irreversible (or high cost to fix)	Cognitive forcing	Mandatory: pre-send checklists, human approval, detailed provenance. Even "Green" drafts need a micro-check.
Agentic	CRM writes, sending email	Partially reversible	Circuit breakers	Mandatory: a confirmation that summarises the action before it runs, and a Stop that tries to cancel or roll back.
Low stakes	Ideation, internal summaries	Reversible (easy to edit)	Light signalling	Optional: confidence badges. Treat these as productivity aids, not safety controls.

Fig. 1 — Friction rises with the risk score: the harder a wrong draft is to undo or to spot, the more the screen makes the reviewer do before it goes out.

For every high-stakes or agentic workflow, run a pre-mortem. Write the headline, "Agent emails a customer a discount nobody approved", and size the blast radius. If the headline is existential, the friction must be mandatory.

Which kinds of hallucination should the review screen catch?

"Hallucination" is too broad a term to design against. Different errors need different mitigations, and as we covered in our post on RAG failures, source-level attacks like poisoning can bypass model-level guardrails entirely.

Fact-conflicting. The model invents a fact. Link each claim to the specific chunk, not a full PDF: the interaction cost is too high and reviewers won't verify.
Source confabulation. The model invents the citation itself. Your backend must verify the link exists and contains the quoted span before showing it. If that check fails, redact the claim. Do not show text supported by a ghost.
Injection and poisoning. The source exists, but its content is malicious. Integrity checks can't catch this; it needs content safety scanning, allow-listed sources and sandboxed tools.
Input-conflicting. The model ignores a constraint, such as "Summarise in 50 words". A disagreement banner flags the drift.

Fig. 2 — Each kind of hallucination needs its own catch. A citation check stops an invented source but does nothing about a real source with poisoned content.

What does a hallucination-aware review screen look like?

You need a UI state machine that changes the screen based on backend reliability signals. Microsoft Research found less reliance on incorrect answers when sources are provided or inconsistencies are shown. To drive it, the backend must return more than a text string. These fields are signals from the orchestrator and evaluators, not truths reported by the model itself.

reliability-payload.json (illustrative)

{
  "faithfulness_score": 0.92,
  "judge_verdict": "PASS",
  "source_integrity": "VALID",
  "policy_flags": [],
  "action_reversibility": "HIGH"
}

faithfulness_score is an estimator, not a truth meter. Gate on coarse buckets (High, Medium, Low) to prevent false precision.

For decision-support tasks, ask the reviewer "What would you send?" before showing the draft. This anchors their own judgement and gives you the data to measure switch fraction. Then the screen takes one of four states:

Grounded (green). High faithfulness, valid sources and a judge "PASS". Each claim has a one-click snippet preview. For high-stakes drafts, require a micro-check that needs content discrimination, such as "Select which of these two snippets supports the claim". This avoids click-through theatre.
Draft (yellow). Lower faithfulness or a judge "UNCERTAIN". Text appears greyed out under a "Draft" watermark, and conflicts are flagged: "This claim conflicts with Source 2." Send stays disabled until the reviewer resolves them.
Abstention (retrieval-only). A judge "FAIL" or invalid sources. Do not generate text, but don't dead-end the reviewer either: show the most relevant snippets and offer "Ask a narrower question" or "Escalate to a person".
Blocked. Policy flags are present (personal data, safety, injection). Stop, explain the trigger and escalate.

Once reviewers learn that "Draft" blocks sending, they will regenerate until they get a stochastic "Green". Set a retry budget: in high-stakes contexts, allow a small, fixed number of regenerations, then force retrieval-only mode or human review.

Fig. 3 — The backend signals pick the screen state, not the model. The retry budget stops reviewers regenerating a Draft until it turns Green.

Who checks the checker?

As we explored in our guide to testing prompts, monitoring is the feedback loop that keeps your state machine honest. Faithfulness scores are just estimators, so you need a human baseline. For each high-stakes use case, a subject-matter expert scores a random sample of production drafts every week against a written rubric. If the judge model and the human labels start to disagree, investigate: model drift, data drift, or a labelling error.

Give whoever owns the agent an anomaly panel that surfaces source confabulation spikes, repeated retry loops and policy near-misses, with a one-click "Kill switch" for specific prompt versions.

What does the EU AI Act require for human oversight?

Article 14 requires that high-risk AI systems come with "human-machine interface tools" that let people monitor them, detect anomalies, override their output and interrupt them, bringing the system to a halt in a "safe state". Since the Digital Omnibus, those obligations apply to Annex III high-risk systems from 2 December 2027 (Regulation (EU) 2026/1744). Most sales and CS agents aren't high-risk under the Act, but Article 14 is still the best blueprint for a review step.

In product terms, Stop must try to cancel pending tool calls, and where the API call has already gone out, trigger compensating actions such as "Undo soft delete". For an agent, the safe state means no further side effects and the operator alerted. For a draft, it means output frozen, audit trail preserved and sending locked until senior review.

How do you know the review step is working?

The CHI 2025 study found that participants became more confident after making wrong reliance decisions in certain contexts, a miscalibration your telemetry must catch (Bo, Wan & Anderson, 2025). Log events such as verify_preview_opened, microcheck_failed, stop_pressed and retry_budget_exhausted, split by risk_class, user_role, judge_verdict, prompt_version and team. Then track three numbers:

Hallucination detection rate. Inject known errors in staging (a "trick protocol") and count how many reviewers catch.
Switch fraction. How often reviewers change their own answer after seeing the draft.
Intervention rate. How often people press Stop or roll back.

Frequently asked questions

What's the minimum viable review screen?

Four things: an abstention state that drops to retrieval-only when confidence is low, micro-checks that require content discrimination on high-stakes drafts, a safe halt that triggers compensating actions for agentic steps, and a retry budget that stops reviewers gaming the system.

Safety is infrastructure

You cannot solve hallucinations with better prompt engineering alone. It requires a shift in how we design the interaction between human and machine, moving from "seamless" to "honest". The interface is the safety system. Calibrated reliance is a design discipline, not a product feature.

Services