Putting a human in the loop is meant to add judgement to a decision. In practice, when the reviewer cannot assess the basis for a system’s recommendation, oversight tends to degrade towards ratification: the person signs off because the system said so, not because they independently verified it. This is automation bias, and it shows up even among trained professionals reviewing decisions they are formally accountable for.
It matters because regulation increasingly assumes it away. Requirements for human oversight, such as Article 14 of the EU AI Act or the safeguards attached to the UK’s automated-decision provisions, presume a reviewer capable of actually disagreeing. A design that presents a recommendation without contrastive information, a confidence measure, or a visible path to override it will not satisfy that requirement in substance, however the process documentation describes it.
It is a different failure from the ironies of automation, which is about a skill eroding from disuse. Automation bias is a trust miscalibration in the moment: the reviewer has the skill but defers anyway, because the interface gives them nothing to disagree with.