Why Nobody Can Hit the Brakes

Listen to this article

Why Nobody Can Hit the Brakes

0:00

The argument, in ten slides

Save it, or send it to whoever sets your AI risk appetite.

Swipe to preview · 10 slidesDownload the PDF

The people who left

In Part 3 I described how the human step in the loop is getting shorter. This part asks why the people who see that most clearly cannot slow it down.

On 8 September 2026 Jacob Coxon, who had spent three years on pretraining research at OpenAI and then Anthropic, resigned in a long post on X. According to Fortune and Time, the post said that both companies were “racing straight to self-improving superintelligence and gambling with our lives,” that “neither company is acting responsibly,” and that the people building AI “earnestly believe that it could kill us all by the end of the decade.” Time reports that the resignation came two months before Coxon’s equity would have vested.

Others spoke up within days. Evan Hubinger, who leads alignment science at Anthropic, wrote: “I personally think it is greater than 10 percent within the next decade.” He added that Anthropic lacks a plan to solve alignment for superintelligence and is not clearly on track to get one. Samuel Marks, who leads Anthropic’s Cognitive Oversight team, posted in a personal capacity that commercial pressure keeps the building going and that researchers only have methods that “nudge AIs towards better behavior.” Time reports two further figures: Marcus Williams, who monitors agents at OpenAI, put the risk at 70 per cent without regulation or a coordinated slowdown, and Geoffrey Irving, a former senior alignment researcher at OpenAI and DeepMind, estimated a 50 per cent chance of human extinction in the coming decade.

Read these with care. They are individual judgements, the definitions differ, and several come through a news summary rather than the original post. Critics have also pointed out that a resignation letter is a public act with its own incentives. I give the cluster weight for a plainer reason. The warnings arrive from people with equity at stake and nothing to sell, and they match what the company leaders say on the record.

What the leaders say

Dario Amodei

CEO, Anthropic

"We must pace the frontier"

Sam Altman

CEO, OpenAI

Agreed with Amodei in public, 12 September 2026

Elon Musk

CEO, xAI

"Dario is right"

Photos: Dario Amodei, TechCrunch, CC BY 2.0, cropped; Sam Altman, Office of the Prime Minister of Japan, CC BY 4.0; Elon Musk, Gage Skidmore, CC BY-SA 4.0, cropped.

Dario Amodei has been the most specific. At an Axios summit in September 2025 he said: “I think there’s a 25% chance that things go really, really badly.” In 2023 he had given a range of 10 to 25 per cent. He dislikes the term p(doom) and put the other side at 75 per cent that things go “really, really well.” Note what the figure means. It is the chief executive of a company that sells frontier models, giving the odds that the thing he sells goes wrong.

On Sunday 6 September, OpenAI’s chief scientist Jakub Pachocki published a post that Bloomberg reported the next day. It quotes him: “This is a time that calls for extreme caution.” And: “I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.” Pachocki added that he was concerned that “no one is prepared for the consequences of a continued rapid rise in machine intelligence.”

Scene 2 of 3 · The odds

The people closest to the systems state real odds. The definitions differ, so the bars cannot be added up.

Evan HubingerAnthropic

More than 10%

AI kills all humans within a decade (his "greater than 10 percent")

Dario AmodeiAnthropic

25%

Things "go really, really badly" (Sept 2025)

Geoffrey Irvingex-OpenAI, DeepMind

50%

Human extinction in the coming decade

Marcus WilliamsOpenAI

70%

Extinction in the next few years, without regulation or a coordinated slowdown

Individual, subjective estimates as reported by Fortune (9 September 2026), Time (15 September 2026) and Axios (17 September 2025). Different events, periods and definitions. Not forecasts and not comparable.

None of these numbers is a forecast in the sense a weather service means it. They are subjective estimates, made under different definitions, over different periods, by people who disagree about what the event is. For scale, Time notes that a 2022 AI Impacts survey found the typical machine learning researcher putting 5 per cent on extinction or similarly severe outcomes. Few boards would accept that for any other technology.

Why one lab cannot stop alone

If the people running the labs believe these odds, why not stop? The structure of the situation answers it.

Scene 1 of 3 · The race

Whatever the other lab does, racing looks better for each lab. Together it is worse for both.

Both slowRisk falls. Positions stay level.

B races, A slowsB takes the lead. A loses market, talent and say.

A races, B slowsA takes the lead. B loses market, talent and say.

Both raceRisk is highest. Neither gains ground.

Schematic of two labs, A and B. The outcomes are described in words, not measured. Structure after Armstrong, Bostrom and Shulman, Racing to the precipice, AI & Society, 2016. My own reading.

Strip the situation to two labs, each choosing to slow or race. If both slow, risk falls and both keep roughly equal positions. If one slows while the other races, the racer leads and the slower lab loses its market, its talent and its say in what comes next. If both race, risk is highest. Each lab does better by racing whatever the other does, which is the structure of a prisoner’s dilemma. Armstrong, Bostrom and Shulman modelled a version of this in 2016 and found that more rivals, and more enmity between them, raise the risk of a dangerous outcome.

Anthropic wrote this reasoning into policy. Version 3.0 of its Responsible Scaling Policy took effect on 24 February 2026 and dropped the earlier implied commitment to pause development when risks were too high. An analysis from the Centre for the Governance of AI quotes the company’s explanation: “the level of catastrophic risk from AI depends on the actions of all frontier developers,” and slowing alone could let less responsible actors lead, which would cost Anthropic its “ability to do safety research and advance the public benefit.”

Amodei’s September essay carries the same logic into geopolitics. He writes that if the United States slows too much, “CCP-associated projects will pull ahead, creating significant national security risk.” He also warns against restraint “in the belief that China will do the same” if it does not. His answer to the verification problem is blunt: any agreement must either have “ironclad verifiability” or be limited enough that defection would not be militarily existential.

I find this argument reasonable and I do not find it comforting. Every actor’s reasoning holds up on its own terms, and the sum of those reasonings is a race.

The brakes people have asked for

Scene 3 of 3 · The brakes

Eight months of attempts to slow down. The race has gone on through all of them.

24 FebAnthropic RSP v3.0 drops its pause commitment

28 Jul"Pacing the Frontier" letter, about 1,300 lab employees; a request for tools, not a pause

8 SepJacob Coxon resigns from Anthropic with a public warning

12 SepAmodei publishes a three-step pacing plan; Altman and Musk agree in public

13-14 SepTrump rejects the call: "Whoever wins AI, wins"

5 NovMETR: Anthropic progress about twice as fast, most key evidence redacted

2026. Sources: Centre for the Governance of AI (RSP v3.0), AI2Work (letter), Fortune and Time (Coxon), darioamodei.com, ITV, Al Jazeera, METR. Trump's date is reported as 13 or 14 September.

The proposals since July share one feature. None asks for a pause.

The “Pacing the Frontier” letter of 28 July 2026 asked the American government to support an international effort to build the technical and governance tools for deliberately pacing automated AI development. Reports put the verified signatories at about 1,300 employees across nearly a dozen labs, with OpenAI and Anthropic endorsing it as companies on the same day. By its own framing it was not a pause. It aimed to build the option to slow before the need arose.

Amodei’s essay of 12 September, “We Must Pace the Frontier,” set out three steps: embedded third-party evaluators with ongoing, employee-like access at every frontier company; common safety standards and “limits on the rate of unchecked AI progress” among labs in democracies; and coordination with authoritarian governments “to the extent this is possible.” Anthropic committed unilaterally to the first step only. Altman and Musk agreed in public that day, in terms that, as one reviewer noted, left the details open.

The politics cut the other way. Trump posted that there was a “SICK conspiracy going on against AI and data centers, and the only one that is happy about it is China,” and, in the same days, that the only guardrails AI needs is “a STRONG AND SMART (High IQ!) PRESIDENT.” House Speaker Mike Johnson said an emergency session to regulate AI would mean losing the race to China. State media in China described the pacing idea as a “Cold War playbook.” Bills to require shutdown capability, risk reporting or a pause were introduced in Congress. I found no report that any has passed.

Dave Karpf’s critique in Tech Policy Press of 17 September is the one I would read alongside the essay. He agrees that an unconstrained race is dangerous. He argues that evaluators paid by the labs they inspect are weak without legal mandates, and that Anthropic, which was preparing an IPO, was proposing rules it could live with. He compares industry-chosen rules to the cigarette industry’s habit of favouring regulation it can shape.

What METR found on 5 November

The most recent evidence is the sharpest test of the idea. On 5 November METR, the nonprofit that evaluates frontier systems, published an independent assessment of AI research acceleration at Anthropic, based on a high-access evaluation from 12 August to 29 October.

METR’s central estimate is that Anthropic is improving the capabilities of its models on well-specified tasks at about twice the rate of March 2024 to October 2025, and it would not be surprised by anything from 1.3 to 3 times. It guesses that “a majority (perhaps ≈80%)” of the acceleration comes from AI itself, and it states that this is a judgement made with substantial uncertainty. It found no convincing bottlenecks to continued rapid progress over the next year. Using all frontier developers’ models, the acceleration is lower, about 1.4 times.

The report also shows the limits of the embedded-evaluator idea. METR says Anthropic redacted a majority of the evidence it found most informative, held unilateral redaction rights, and cancelled a compute-allocation check partway through. “The public has to rely on METR’s judgment,” it wrote, “much more than would be ideal.” Anthropic’s co-founder Sam McCandlish said that sharing the exact drivers of the speedup could give others a roadmap to move faster.

Take this as a fair reading of where pacing stands. A lab opened its books to an outside evaluator, which is real progress. It then kept the commercially sensitive parts back, for the same competitive reason that makes pacing hard. The race shapes even the brake.

What coordination has worked

Nothing in the history of technology makes this easy, and a few cases show what helps.

In February 1975 around 140 biologists met at Asilomar in California and agreed rules for recombinant DNA research, after a voluntary moratorium the year before. It held because the community was small, the risk was concrete, and the researchers shared a profession. The Montreal Protocol of 1987 phased out ozone-depleting chemicals and is widely regarded as the most successful environmental treaty. Harm could be measured, substitutes could be built, and trade rules penalised countries that stayed out.

AI has had the opposite results so far. The Future of Life Institute’s open letter of March 2023 asked for a six-month pause on training the most powerful systems, and training went on. In May 2024 sixteen companies signed the Frontier AI Safety Commitments at the Seoul summit, and those commitments are voluntary, with no mechanism to enforce them. The pattern I draw from these cases is my own judgement. Coordination holds when the harm can be measured, when a substitute exists, when defectors pay a price and when the parties can verify each other. Today’s frontier race has the first and the fourth in weak form, and none of the other two.

What one organisation can decide

You cannot end the race. You can decide how you behave inside it, and that decision is more valuable than it looks, because you are probably a buyer of frontier AI and a buyer has leverage a lab does not.

My own framework. The February example refers to Anthropic’s RSP v3.0, discussed above.

None of this stops a rival lab. It does change what the labs find worth competing on, and in the cases above coordination started with participants who made their own rules first.

Questions for leaders

1. What would our organisation refuse to deploy, written down, and who can say no when a competitor has already done it?
2. What do we require a vendor to show about safety, and who checks it independently?
3. How do we find out when a vendor changes a safety commitment, and what is our plan if it does?
4. If a model generation arrived this quarter that was twice as capable, who decides how fast we adopt it?
5. Whose view on slowing down do we hear, and is anyone in the room paid to disagree?

Part 3

Agent Swarms and the Closing Loop

The loop that makes the race faster, and the checkpoints to keep.

Part 2

When Machines Prove What We Cannot Follow

What a machine check covers when no person can follow the proof.

Part 1

The Takeoff Nobody Noticed

The speed mismatch that sets up everything in this part.

Part 5 takes up Stuart Russell’s gorilla problem: what happens to the less capable species when a smarter one has other priorities. If your board is writing its own refusal line for AI, that is a conversation my executive programme is built for.