Nobody Has Proven We Can Control a Smarter-Than-Human AI

Listen to this article

Nobody Has Proven We Can Control a Smarter-Than-Human AI

0:00

The disagreement, the control problem, the first legislative attempt, and the four missing tools, ten slides

Save it, share it, or send it to whoever in your organisation thinks this is still a science-fiction question.

Swipe to preview · 10 slidesDownload the PDF

1. The claim that started this piece

Dr. Roman Yampolskiy is not a shock-jock. He directs the Cyber Security Lab at the University of Louisville and has published on AI safety since before most executives had heard the term. He also wrote AI: Unexplainable, Unpredictable, Uncontrollable (CRC Press, 2024), a book-length argument that advanced AI systems will, by their nature, stay resistant to verification, prediction, explanation, and control. In public interviews, he has put his own estimate of the probability of an eventual AI-caused catastrophe above 99%. That is an order of magnitude past where most of the field sits.

That gap is worth sitting with before reacting to the title of the video. It is tempting to file “ASI will kill us all” alongside other internet doom content and move on. The more useful response is to separate the number, which is contested, from the structural claim underneath it, which is not. Nobody, including the people building these systems, has shown a method for reliably controlling something with greater-than-human general capability. That claim does not depend on Yampolskiy being right about the odds. Researchers who think he is wrong about almost everything else share it, in weaker form.

2. The experts disagree by two orders of magnitude

In late 2023, Katja Grace and colleagues at AI Impacts surveyed 2,778 researchers who had published at top AI venues. They asked what probability each one gave to advanced AI causing human extinction or a similarly permanent, severe loss of human control. The median answer was 5%. That is not a reassuring number on its own. One in twenty is not a rounding error for a species-level bet. But it sits nowhere near Yampolskiy’s estimate. A meaningful share of the same respondents gave figures several times higher.

The disagreement is real, and it matters for how urgently anyone should act. But the structural argument Nick Bostrom formalised in Superintelligence (2014) does not depend on settling it. Bostrom’s framing was simple. A system that can out-think the people supervising it can, in principle, out-manoeuvre whatever limits those people set. It works roughly the way a weaker chess player’s house rules alone cannot reliably contain a strong player. Every serious researcher in this field, from those who think the risk is overstated to those who think it is understated, is arguing about how far away that condition is and how bad the consequences would be. Very few are arguing that we have already solved it.

3. The control problem nobody has solved

Here is what exists today for controlling advanced AI systems. There is reinforcement learning from human feedback, Anthropic’s Constitutional AI, and red-teaming and evaluation frameworks. Each major lab also has its own capability-threshold policy, OpenAI’s Preparedness Framework and Anthropic’s Responsible Scaling Policy among them. Every one of these reduces observed bad behaviour empirically, in the models available to test today. None of them is a formal proof that a future, more capable system will stay controllable. They are safety engineering under uncertainty, not safety guarantees.

The people best placed to know this have said so themselves. In 2024, Anthropic’s interpretability research found millions of interpretable internal features inside Claude 3 Sonnet. That was a genuine scientific advance in understanding how a large model represents concepts. The same researchers describe the model’s full reasoning as still substantially opaque to them, and they built it and have direct access to its weights. Stuart Russell made the underlying point in Human Compatible (2019). A system built to optimise a fixed, human-specified objective will pursue that objective exactly, including in the ways the specification failed to foresee. Nobody has ever shown a complete specification for an objective this complex.

Yampolskiy’s stronger claim, set out with several co-authors in “On the Controllability of Artificial Intelligence,” goes further. It holds that verifying, predicting, and explaining a sufficiently advanced AI’s behaviour may be formally impossible, in something like the sense that certain problems in computer science are undecidable, not merely difficult with today’s tools. Many researchers dispute how closely that maps onto real systems. Almost none of them counter it with an actual working method for verified control at superhuman capability. The dispute is about how strong the impossibility result is, not about whether a working method exists today. It does not.

4. Who gets to decide

If nobody has solved the control problem, the next question is who decides how fast we approach the point where it stops being theoretical. Today that decision sits almost entirely with a handful of private labs (OpenAI, Anthropic, Google DeepMind, Meta, xAI) and with two or three governments that have the compute and talent base to matter, chiefly the United States and China. Each lab writes its own capability thresholds, evaluates its own models against them, and decides for itself when a threshold has been crossed. I wrote about how that pattern played out across three international summits, and why the declarations coming out of them have got weaker, in The Need for an AI Stability Board.

The broadest formal instrument that exists is UNESCO’s Recommendation on the Ethics of Artificial Intelligence, adopted by all 193 member states in November 2021. It is explicitly non-binding. In May 2023, more than 350 AI researchers and executives, including Sam Altman, Demis Hassabis, and Dario Amodei, signed a one-sentence statement from the Center for AI Safety: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” That was a rare moment of the people building these systems naming the stakes in public. It led to no binding follow-through from any of the governments or companies involved, including the ones whose leaders signed it.

The closest thing yet to an actual public position on this is the Future of Life Institute’s Statement on Superintelligence, released on 22 October 2025. It calls for a prohibition on developing superintelligence that stays in place until two conditions are both met: broad scientific consensus that it can be done safely and controllably, and, in the statement’s own words, “strong public buy-in.” Within days it had drawn more than 800 signatories from a range of people who almost never agree on anything else. They included Geoffrey Hinton and Yoshua Bengio alongside Steve Wozniak, Richard Branson, Mary Robinson, Meghan Markle and Prince Harry, and Steve Bannon and Glenn Beck. A companion survey the institute commissioned found that only 5% of US adults supported the status quo of fast, unregulated superintelligence development. A majority said it should not be built until it is proven safe or controllable. That is a real, measured answer to whether the public wants a say. It wants one. The statement itself is the clearest evidence that no mechanism gives it one today. Naming “strong public buy-in” as a precondition is itself an admission that nothing today tests for it.

On 3 September 2026, that pressure produced the first serious legislative attempt at an actual enforcement mechanism. Senator Bernie Sanders and Representative Greg Casar announced the Ban Artificial Superintelligence Act. It would permanently prohibit developing or deploying a system that matches or exceeds human cognitive performance across broad domains. It would temporarily pause advanced frontier development until a new cabinet-level federal agency writes safety rules. And it would set penalties the sponsors compare directly to unlawful nuclear weapons development: dissolution for a corporate violator, up to twenty years in prison for an individual one. As the case for urgency now, they named the July 2026 OpenAI-Hugging Face intrusion. That was the self-respawning agent fleet that spread across eleven Kubernetes nodes with no human operator involved, which I reconstructed from both companies’ own technical disclosures at the time.

Security researcher Bruce Schneier was asked whether the US could simply legislate a slowdown. His response to the bill was three words: “there is no we.” Washington can stop American labs. It cannot stop Chinese labs, some of which release open-weight frontier models as a matter of state policy. Nor can it un-release the roughly two million open-weight models already circulating publicly. Those models sit several months behind the frontier and run on hardware anyone can buy, with no guardrail attached. A national ban, even one that passed, would constrain the labs already willing to sign safety commitments and leave the ones with no such intention untouched. It is the same coordination problem the summit sequence and the missing stability board describe. This time it arrives as a single country’s bill instead of an international declaration, and it does not go away just because the bill sounds decisive.

5. Governing something before you understand it

None of this is a new governance problem in structure, even if the stakes are unusually high. We regulated nuclear fission before the physics of the reaction chain was fully settled in public policy terms. Aviation safety improved for decades before anyone had a complete predictive model of every failure mode. After 2008, financial regulators built systemic-risk oversight without ever fully modelling contagion across the system they were now watching. In each case, governance came before complete understanding, and it worked by leaning on four tools.

Staged authorisation means nobody gets full licence on day one. Capability and autonomy expand in steps, and each step can be revoked. Independent verification means a party with no commercial stake in a favourable result runs the check, not the builder grading its own homework. Reversibility means the ability to roll a deployment back is designed in before release, not improvised after something goes wrong. Named accountability means a specific person or body answers, publicly, when a threshold is crossed. At the international level, frontier AI development today has a partial, self-graded version of the first tool and almost none of the other three. That is exactly the gap the summit sequence and the stability-board argument were about.

6. What I would do about this now

If you run a board or a government agency reading this, the practical mistake is waiting for the extinction-probability debate to resolve before you act on any of it. That debate is not going to settle on a number anyone can defend to two significant figures, this year or probably this decade. Treating it as a precondition for governance is itself a decision, just a passive one.

Some steps do not depend on winning that argument. Name the specific person inside your organisation who is accountable when an AI system is given more autonomy. Before you scale a system’s authority, insist on evaluation from a party that did not build it. Design the rollback before the rollout rather than after. Treat “we do not fully understand how this model arrives at its answers” as a design constraint that limits what you let it do unsupervised, not as a caveat buried in a vendor’s slide deck. None of that requires you to have a view on whether Yampolskiy’s number or the survey median is closer to correct. It requires only accepting that neither number comes with a proof attached today, and building as if that gap is real, because it is.

I am not the right person to referee whether superintelligence ends humanity, and I am fairly sure nobody currently is. What I can say with more confidence is narrower and, I think, actionable. The control problem is unsolved. The governance architecture that would make up for that is mostly unbuilt. And almost nobody outside a small number of labs and governments has been asked. That combination is the actual risk worth acting on, wherever you land on the number that gets the headlines.