Automation bias

What is automation bias?

Automation bias is what happens when a person stops checking and starts agreeing. A system produces an answer, a recommendation or a draft, and the person accepts it even though their own judgement or something on the screen says otherwise. The system did not force anything. The person handed over the decision because the system had usually been right.

It comes in two shapes. An error of commission is acting on a wrong recommendation: the system says approve, so you approve. An error of omission is missing something because the system stayed quiet: no alert came, so you assumed there was nothing to see. The second one is harder to catch, because nothing happened that you could have reacted to.

The term comes from human factors research on cockpits and control rooms in the 1990s, and moved into medicine as decision support systems started suggesting diagnoses. Since 2023 it has become a live problem for any company that puts a language model in front of staff, because the same reflex applies to a drafted mail, a proposed journal entry or a suggested answer to a customer.

The reviewer who has approved 400 credit notes without finding one problem is not really reading the 401st. That is automation bias, and it is not a character flaw. It is the normal way a person behaves next to a system that is right most of the time.

How automation bias arises

Four things push a person from checking to agreeing, and they usually arrive together.

Checking is work, agreeing is free. The researchers who first named the effect described it as using the automated cue as a shortcut instead of actively looking for and weighing information. A brain that can skip a step will skip it, and a recommendation that already looks like an answer is exactly that kind of step.

Trust builds on the last hundred results. Every correct output makes the next one easier to accept without looking. That is reasonable behaviour, right up to the case where the system is wrong, and by then the habit of not looking is in place. The better a system performs, the less prepared its users are for its failures.

Time pressure and a second task. A review of the research in the journal Human Factors in 2010 found that complacency towards automation shows up mainly when the operator has other tasks competing for attention. A reviewer with a queue of sixty items and a meeting at eleven is in exactly that situation.

Overriding costs more than agreeing. If a wrong override is discussed and a wrong approval is not, people learn to approve. In the cockpit studies from the late 1990s, pilots who felt personally accountable for how they worked with the automation double-checked it more often and made fewer errors. The same studies found that pilots tended to remember having seen confirming cues that were never on the screen.

Two findings from that older research matter for anyone hoping to train the problem away. The 2010 review concluded that both complacency and automation bias hit novices and experts alike and persist despite training. A study on flight crews found that training aimed at automation bias reduced commission errors but did nothing for omission errors. People learn to question what the system says. They do not learn to notice what it did not say.

Why language models make it worse

Classic decision aids gave a score, a flag or a category. A language model gives you a paragraph, written the way a competent colleague would write it, addressed to you, in your own context. Three properties of that output make the bias stronger.

The tone is the same for right and wrong answers. A model that has just invented a reference writes it in exactly the voice it uses for a correct one. There is no wobble, no hesitation, nothing on the surface to react to. The hallucination entry covers what goes wrong inside the model. Here the point is what it does to the reader: the usual signals that tell you a colleague is unsure are missing.

The answer is personalised. It uses your customer's name, your product codes and the figures you gave it. A generic warning is easy to ignore. A draft that already reads like something you would have written yourself is very easy to send.

The model agrees with you. Sycophancy, the tendency of a model to follow the view you signalled, means the output often confirms what you already thought. Confirmation from a fluent, confident source is the version of automation bias that is hardest to notice from the inside.

The US standards body NIST wrote this into its generative AI profile in 2024: over time people may over-rely on generative AI systems or perceive their content as higher quality than content from other sources, which it names as an example of automation bias, and it adds that this over-reliance makes the other risks worse, confabulation included. The International AI Safety Report of 2026, written by a group of researchers for the governments backing it, noted that evidence for automation bias in tasks assisted by generative AI has accumulated since the previous edition.

What recent studies found

The pattern is consistent across settings, and three studies put numbers on it.

Radiologists, 2023. A team at the University Hospital of Cologne had 27 radiologists read 50 mammograms with an AI aid that suggested a category. When the suggestion was right, readers at every experience level scored around 80 percent. When the suggestion was wrong, the least experienced readers dropped to about 20 percent correct and the most experienced to about 45 percent. Experience softened the effect. It did not remove it.

Physicians with AI training, 2025. A randomised trial run in 2025 and published in NEJM AI followed 44 physicians who had all completed a 20-hour AI literacy course. Half got a language model with correct suggestions, half got one that was deliberately wrong in three of six cases. The group with the flawed assistant scored about 73 percent on diagnostic reasoning against about 85 percent for the other group, a drop of some 14 points after adjustment. Consulting the model was voluntary. The training did not protect them.

2,784 people on a data task, 2026. A study published in the Harvard Data Science Review gave participants a data extraction task with AI suggestions of varying quality. When correcting a flagged suggestion took extra effort, people accepted more wrong ones. The strongest predictor of how well someone did was not their background but their attitude towards AI: the sceptics caught more errors.

Put together, the three say something uncomfortable for anyone rolling out an assistant. Experience helps a bit, a training course does not help much, and adding a step to the correction path makes things worse.

Two cases from an ordinary week

The reviewer of credit notes. An agent prepares credit notes for damaged deliveries and a person approves them. For the first month the person opens each one, compares it with the delivery note and finds nothing wrong. By the third month the review is a glance and a click. In month five the agent credits the full invoice for a partial return because the return note listed the wrong line, and the approval goes through in four seconds. Nobody skipped a step in the process document. The step just stopped meaning anything. The human-on-the-loop entry works out how to set up that queue so the review stays real.

The sales rep and the quote. A rep asks the assistant to draft a quote for a returning customer. The draft is polished, uses the right product names and last year's prices, and states thirty days payment terms. This customer is on prepayment after two late invoices, which the rep knows and the assistant did not. The quote goes out unread because it looked finished. That is an error of commission, and the trigger was the quality of the prose, not a lack of knowledge.

Automation bias versus sycophancy

The two are easy to confuse, because both end with a person and a machine agreeing on a wrong answer. The difference is who bent.

Automation bias is the person bending to the machine. The model said something, and you took it over your own judgement or over the evidence on the screen. The fix sits on the human side: the review process, the incentives, the design of the screen.

Sycophancy is the machine bending to the person. You signalled a preference, and the model shaped its answer to match. The fix sits on the prompting and training side: take yourself out of the question, ask for the case against.

They feed each other. You state a view, the model agrees with it in confident prose, and you accept that agreement as an independent confirmation. Two parties bending towards each other produce a decision nobody actually made. The tell is a fast, comfortable conversation that ends exactly where you started, and the cure for that is the same in both entries: ask the question again with your preference removed.

What helps against automation bias, and what does not

The measures that work all change the task or the incentive. None of them ask the reviewer to try harder.

1. Show the evidence and the confidence next to the answer. A draft that shows which source lines it used, and where the model was unsure, gives the reviewer something to check instead of something to accept. A bare answer can only be agreed with.

2. Make overriding cheap and agreeing costly at the moments that matter. Approving a 15,000 euro credit note should take a click more than approving a 40 euro one, and rejecting should never take more steps than approving. The 2026 study above found that extra effort on the correction path is enough to make people accept more errors. Remove the effort there.

3. Inject known errors. Put a deliberately wrong item in the queue now and then and see whether it gets caught. This is the same idea as the mock phishing mail. It tells you whether the review is real, and it reminds reviewers that wrong items exist.

4. Measure the override rate. Count how often a reviewer changes or rejects what the system proposed. A rate near zero for months is not proof of a perfect system. Track the time per review as well: a time that keeps shrinking means the same thing.

5. Rotate reviewers. A fresh reviewer has not yet built the trust that makes them stop looking. Swap the person on the queue every few weeks and let the incoming one start with a full read.

6. Keep the person doing part of the task. Someone who only watches loses the skill and the attention to intervene. Let the assistant draft and let the person write the one paragraph that matters, or let the agent book and have the person handle every case above a threshold by hand. Doing keeps the judgement warm. Watching does not.

Three things do not work, and they are the three most companies reach for first. A disclaimer under the answer, of the kind that says the model can make mistakes, is read once and never again. A training slide on automation bias produces the result the flight crew study and the physician trial produced: people know about the effect and still show it. And asking staff to stay critical hands the problem back to the person whose attention is the scarce resource, with nothing added to help them.

The EU AI Act takes the same view. Article 14(4)(b) requires that whoever oversees a high-risk system is able to remain aware of the possible tendency of automatically relying or over-relying on the output, which the text names as automation bias, and it singles out systems that give people information or recommendations to act on. Awareness is a design duty on the provider there, not a poster in the break room. The human oversight entry works out what that article asks of you as a deployer. For every assistant that is not high-risk, which is nearly all of them in a Belgian company, the six measures above are the practical version of the same rule.