TLDR: If your AI system has a human review step and an approve button, you probably have automation bias, not oversight. The fix isn’t more training - it’s testing whether humans are actually reviewing by embedding things they should catch.

How the queue trains the reviewer

An LLM or rules engine generates outputs at volume. A human is placed in the loop to review before anything ships. The interface surfaces a suggestion and asks: accept or reject?

Within a few weeks, approval rates are hovering near 95%. The queue has a backlog. Accept gets faster.

And the asymmetry matters. A 2025 study (Beck, Eckman et al.) ran 2,784 participants through an AI verification task and found that when flagging an error required the extra step of typing a correction, participants accepted more wrong answers, not fewer. The act of rejection - staring at what the model produced and then having to write something different yourself - is cognitively harder than moving on. Making rejection harder pushed people toward approval. The interface isn’t neutral. It has a thumb on the scale.

This shows up in caseworker notes, AI-assisted eligibility determinations, clinical decision support tools - any system where “accept” is one click and “reject” means stopping, forming a different answer, and documenting why. The path of least resistance is always approve.

This has a name

It’s called automation bias. Mosier & Skitka documented it in 1996 - humans systematically over-rely on automated recommendations, especially under time pressure and high volume. The effect compounds when the system has been accurate before: trust builds until a bad output slips through unnoticed.

Clinical decision support in healthcare is the most documented version of this. Alert override rates in EHR systems range from 49% to over 90% depending on alert type - clinicians click through because the volume of low-signal alerts has trained them to dismiss everything quickly. The high-signal alert that actually matters looks identical to the noise.

Medicaid prior authorization is reportedly similar - AI systems generating coverage recommendations at scale, human reviewers approving at high rates under volume pressure. Whether that constitutes meaningful oversight is a question litigation is beginning to force.

It gets worse the longer you work there

New hires read the manual. Veterans trust their muscle memory.

This is a working theory more than established research, but: when a rules engine is updated or a model is retrained, the people most likely to catch behavioral drift are the ones who haven’t yet built the habit of clicking through. The tenured reviewer who knows “how these usually come out” is the least equipped to notice when they don’t anymore.

Oversight quality may actually degrade as familiarity increases - which is the opposite of what teams usually assume.

The red herring test

Embed known-wrong outputs into the review queue and measure the catch rate.

An LLM-as-judge that knows the correct answer seeds the queue with a small percentage of deliberate errors - a citation that doesn’t support the claim, a rule that contradicts the source document, a classification that’s clearly off. The human reviewer doesn’t know which items are tests.

If your reviewers are catching the red herrings, your oversight is real. If they’re approving them at the same rate as everything else, you have a green button, not a review process.

The goal isn’t catching individual reviewers. It’s an honest measurement of whether the oversight is real - and a feedback loop for fixing the interface, queue size, and consequence design before something ships that shouldn’t.

The accountability gap

When a rubber-stamped output causes harm - a wrongly denied benefit, a misclassified record, a code vulnerability that shipped - who owns it?

The AI vendor says the human approved it. The human says they were given 200 items and 20 minutes. The team that built the interface says the process was followed.

Nobody has a clean answer to that yet. The red herring test doesn’t resolve the accountability question - but it does force the more basic one first: was there ever real oversight to begin with.

Related reading

• Bias in the Loop: How Humans Evaluate AI-Generated Suggestions - Beck, Eckman et al. (2025)
• These Aren’t the Reviews You’re Looking For: How Humans Review AI-Generated Pull Requests (2026)
• Automation and Human Performance: Theory and Applications - Parasuraman & Mouloua (1996), source of the original Mosier & Skitka chapter

Thinking through human review design for an AI system in a high-stakes context? Reach out.