Executive Summary
Most modern AI systems are machines that recognise patterns. They are not autonomous minds. Most AI disappointments and incidents start when people treat them as if they were, through myths about autonomy, functionality, objectivity, and inevitable ROI.
Core conclusions
- Four fallacies distort decisions. People assume the system can run on its own, that a deployed system must work, that precise output must be objective, and that adopting AI is enough to guarantee ROI.
- For governments and infrastructure operators, these myths create risk across the whole system, well beyond one failed project. Liability becomes unclear, policy drifts, and the institution slowly loses its own judgment.
- The fix is technical honesty. Design for errors and for handing cases to a person. Check that the problem makes sense before you deploy. Build accountability into every stage of the model’s life.
The four fallacies, in ten slides
Save it, share it, or send it to whoever is writing your AI governance policy.
The main fallacy about artificial intelligence has little to do with whether it “does not work”. People routinely treat AI as autonomous, intelligent and dependable in ways its maths and engineering do not support.
AI’s Pattern Machinery Gets Mistaken for Intelligence
Most modern AI systems are high-dimensional pattern recognisers. They are not general intelligences. They learn from historical data how to map inputs to outputs. They do not understand, intend, or grasp the situation around them the way a person does.
Several consequences follow:
- Apparent “reasoning” is often interpolation between patterns the model has already seen. That is why models fail abruptly when the data they meet drifts from the data they learned.
- Precision is an illusion. A model that is too precise on its training data is usually overfitting. It has not learned the general structure.
- Any creativity or meaning people see in the output is something they project onto it. The output comes from statistical processes over training data.
Treating this pattern machinery as an intelligent colleague or an autonomous decision-maker is the central fallacy. It sits under many AI disappointments and incidents.
Four Specific Myths That Distort Decision-Making
- The Fallacy of Autonomous AI
A common story says AI systems can act as autonomous agents and reliably handle complex social tasks with little supervision. In practice, even advanced models struggle when social situations change, when the environment keeps shifting, and with the unwritten rules people follow every day.
- Studies show models do poorly at reading real-world social situations that involve several people. That skill matters in fields such as autonomous mobility, care, and public safety.
- Systems sold as “autonomous” often hide a lot of human work: data labelling, prompt engineering, escalation paths and operational oversight.
Calling these systems autonomous invites people to hand over too much responsibility. It weakens human accountability where safety is at stake. The fix is a disciplined screen for which tasks should get autonomy at all: see Critical Tasks Suitable for Agentic AI.
- The Fallacy of Functionality
Another lasting myth says that once an AI system is deployed, benchmarked and built into workflows, it must work for its stated purpose. Evidence from real deployments shows you cannot rely on that.
- Many AI systems are “broken by design” because the thing they try to measure has no defensible signal you can measure. Examples include attempts to infer “criminality” or personality traits from facial images.
- In these cases, no amount of model optimisation can make the system valid. The way the problem is defined is unsound from the start.
The practical risk is that teams treat the outputs as if they measured something real they can act on. They then build policy and operations on a signal that does not exist.
- The Fallacy of Precision and Objectivity
Many people assume AI is a precise, objective technology that will beat human judgment through sheer scale and computing power.
- In reality, robust models are built to tolerate error and noise. Too much precision usually means the model has overfitted to historical data. That makes it less robust when the real world varies.
- The data limits how objective a model can be. Models inherit historical biases and gaps, and often amplify them. This has been shown again and again in hiring, credit, and criminal justice.
Over-trusting AI’s precision pushes teams to automate decisions that need context, a way to challenge them, and a stated reason.
- The Fallacy of Inevitable ROI
In boardrooms and policy papers, AI is often presented as inevitable: deploy it and productivity gains will follow. The adoption data suggest otherwise.
- Surveys show that most organisations report some AI experimentation. Only a small fraction reach mature, scaled use with measurable value.
- The gap is mostly not about the technology. It comes from weak problem selection, too little redesign of how work runs, too little spent on data quality, and poor change management. I break down these specific failure patterns in Why Most AI Projects Fail.
If you assume that adding AI to existing processes guarantees productivity, you get scattered pilots and inflated expectations. Disappointment follows when structural limits decide the outcome.
Why This Fallacy Is Dangerous for Governments and Infrastructure Operators
For public authorities and critical infrastructure operators, these myths turn straight into risk across the whole system.
- Safety and liability: When no one clearly owns an AI-assisted decision, a grey zone opens up. Neither the vendor nor the operator fully owns the failure, which makes redress harder when harm occurs.
- Policy distortion: Leaning too hard on AI-generated insights can pull policy away from ground truth. This is most likely where data is scarce or biased (informal economies, under-represented communities, emerging risks).
- Capability hollowing: If you hand too many decisions to AI, you risk losing the analytical, operational and institutional memory your people hold. You then cope less well when systems fail or conditions change.
In a smart city, this can show up as over-automated control rooms, brittle demand forecasts, or overconfident risk scores in policing and social services. All of these wear down citizen trust once people see the errors.
Towards a More Technically Honest AI Agenda
The alternative to the fallacy of AI is technical honesty about what these systems can and cannot do. That is a different thing from pessimism about technology.
A more rigorous stance could include:
- Treating AI as probabilistic infrastructure, not autonomous intelligence: design it to tolerate errors, degrade gracefully, and pass cases clearly to a person.
- Starting from construct validity, not model availability: only use AI where the signal you want makes sense, can be measured, and is legitimate for the institution to use.
- Embedding accountability by design: name who is responsible for model performance, data governance and decision outcomes at every stage, including decommissioning.
- Measuring value in system terms: look at how reliable, fair and resilient the whole service is from end to end.
From fallacy to technical honesty
Technical honesty designs for bounded reliability: the system can be wrong, so someone owns the error and a human can step in.
Schematic summary of the four fallacies and the corrective practices set out in this post.
AI is a useful pattern infrastructure with bounded reliability, not a substitute for institutional judgment. Treating it otherwise is not just a technical error; it is a governance failure.
Free tool
TRACE Agent Evaluation
Score a candidate task on Traceability, Reversibility, Acceptance, Compliance and Escalation before you assume it’s ready for autonomy.
Free tool
ISO 42001 AIMS Readiness Checklist
Check whether accountability is built into your AI lifecycle, or just assumed the way this piece warns against.