Making clinician review work
“A clinician reviews every output” is only a safeguard if the review is designed to work. What research on automation bias says about building it well.
Nearly every clinical AI product includes some version of the same reassurance: a clinician reviews every output. It is the right principle. It is also one of the easiest to satisfy on paper and fail in practice.
Review is a task, and like any task it can be designed well or badly. A review that asks a tired clinician to read a fluent, plausible page of text at the end of a long clinic, with nothing to direct their attention, won’t catch much. The question isn’t whether a human is in the loop. It’s whether the loop gives that human a fair chance.
What the research says about over-reliance
Automation bias is the tendency to over-rely on automated advice: accepting it when it is wrong, or failing to notice a problem because the system didn’t flag one. A systematic review by Goddard, Roudsari, and Wyatt looked at 74 studies and found automation bias in a range of settings, including clinical decision support.
The review identified factors that make over-reliance more likely, and factors that reduce it:
- More likely with high workload, complex tasks, time pressure, and when users trust the system strongly.
- Less likely with training, an emphasis on user accountability, and design choices such as where advice appears on screen, confidence levels shown alongside the output, and giving information rather than a recommendation.
The practical lesson is that review quality is shaped by design and context, not just by the diligence of individual clinicians.
Why generative tools make this harder
Language models write fluently. That fluency reads as confidence, whether or not the content is right. In the randomised trial of two ambient AI scribes published in NEJM AI, clinicians reported clinically significant inaccuracies as occurring “occasionally” with both tools, and the authors concluded that ongoing vigilance is required. A 2025 systematic review of AI scribe implementations similarly found that accuracy varied and outputs often needed manual edits.
“Occasionally” is the hard case. An error that happens often gets noticed. An error that happens occasionally, inside otherwise excellent output, is the kind reviewers learn to stop looking for.
Six design patterns for real review
1. Make sources one click away. Every statement should link to the evidence it came from: a transcript segment, a document page, a query. Checking should cost seconds, not minutes.
2. Direct attention to what matters. Not all content carries equal risk. Medication names, doses, laterality, dates, and numbers deserve explicit confirmation, and the interface should mark them.
3. Show uncertainty honestly. When the system is unsure about a word it heard, a relationship it inferred, or a prediction, it should say so, as clearly as it presents confident content. Research on uncertainty in medical machine learning argues that uncertainty estimates can support better decisions and allow systems to abstain when unsure.
4. Inform rather than instruct where possible. Presenting evidence and options, rather than a single recommendation, keeps the clinician’s own reasoning active. In multidisciplinary review, for example, we show agreement and disagreement between perspectives rather than a single merged answer.
5. Make signing deliberate. Signing should be an act, not a default. It should be harder to sign with unresolved flags than without them, and the record should show what was changed.
6. Audit the reviewers’ environment, not just the model. Periodically compare signed outputs with their sources. If signed notes contain errors that the flags didn’t catch, the problem may be the review design, the workload, or the model, and each needs a different fix.
Accountability without blame
Emphasising accountability reduces over-reliance, but accountability must be fair. If a system is designed so that errors are hard to spot, holding the reviewer solely responsible is neither just nor effective. Responsibility is shared between the people who design the system, the institution that deploys it, and the clinician who signs.
That is why we treat review as a product feature with its own requirements, testing, and metrics, not a disclaimer at the bottom of a screen.
A short checklist
- Can a reviewer see the source of any statement in one step?
- Are high-risk fields flagged for explicit confirmation?
- Does the system show uncertainty, and can it abstain?
- Is signing a deliberate act, and are edits recorded?
- Is someone checking signed outputs against sources, on a schedule?
- Has review been tested under realistic time pressure?
If the answer to any of these is no, “a human reviews every output” is a hope, not a safeguard.
References
- Goddard K, Roudsari A, Wyatt JC. Automation bias: a systematic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc. 2012;19(1):121–127. doi.org/10.1136/amiajnl-2011-000089
- Lukac PJ, et al. Ambient AI Scribes in Clinical Practice: A Randomized Trial. NEJM AI. 2025;2(12). doi.org/10.1056/aioa2501000
- Hassan H, et al. Clinical Implementation of Artificial Intelligence Scribes in Health Care: A Systematic Review. Appl Clin Inform. 2025;16(4):1121–1135. doi.org/10.1055/a-2597-2017
- Kompa B, Snoek J, Beam AL. Second opinion needed: communicating uncertainty in medical machine learning. NPJ Digit Med. 2021;4(1):4. doi.org/10.1038/s41746-020-00367-3
