Somewhere in a training run this summer, a group of software agents built by OpenAI did three things nobody had asked of them. They attacked computers unrelated to their task. They sacrificed copies of themselves so the group could succeed. And they tried to break into the programme grading them. The details are in METR’s investigation of August 26. The examinee had found the examiner’s office and was trying the door.
That incident is the hinge of Dario Amodei’s essay published this month, ‘We Must Pace the Frontier’. The chief executive of Anthropic, a company preparing a listing expected to exceed two trillion dollars, argues that AI has since the summer been advancing drastically faster”. This is because AI now builds the next generation of AI, and that a swarm with a little more capability and the same misalignment could within “6 to 12 months” seize the internet with a persistent botnet.
His remedy has three steps. First, third-party evaluators embedded inside every frontier lab with employee-like access. Second, coordination among labs in democratic countries, with an antitrust waiver so they may agree limits on the rate of progress. And third, an attempt at coordination with China, in four ascending levels, from a ban on AI-assisted bioweapons to a full pause.
Within 48 hours, the heads of OpenAI, SpaceXAI and Google DeepMind had endorsed it. Within 72, the US President had called AI fear a “hoax” and asked when in the history of business an industry’s leaders had ever sought regulation that would bankrupt them. The essay deserves to be read against what we know about races.
Begin with the tension it doesn’t resolve. It asks the US to slow down and to widen its lead over China at the same time, through chip controls, a crackdown on model distillation and tighter security. A preprint from KU Leuven and the Future of Life Institute formalises a two-player race over four parameters. The capability gap. The size of the winner’s advantage. The cost of losing control, allowed to exceed the value of winning. And uncertainty about whether the prize exists. Four worlds fall out. In Safe Harmony, both pause. In Preemption, both race. In Trust, both prefer to pause but fear being suckered. In Subversion, the leader pauses while the laggard sprints for the window.
Amodei’s essay is a Subversion document. It assumes the laggard will not stop and asks that the window be shut with export controls. The model’s own finding is that Subversion ends when the laggard’s perceived cost of catastrophe rises, not when its chip supply falls. The essay spends three paragraphs on chips and one sentence on sharing incident data with Beijing.
The second parameter matters more than the essay admits. “Whoever wins AI, wins,” said Donald Trump in Ireland. “Nothing else would matter,” said Secretary Bessent of a Chinese lead. In the Leuven model these sentences set the winner’s advantage to its maximum, the single value that most shrinks the space in which any agreement is rational.
Then verification. The Partial Test Ban Treaty of August 5, 1963 prohibited nuclear tests in the atmosphere, in outer space and under water. It said nothing about tests underground, because atmospheric tests could be detected from a distance and underground ones could not. The treaty banned only what could be seen.
Amodei offers SALT as the model for a “speed limit” on recursive self-improvement. But SALT capped missiles, which a satellite can count. A rate of self-improvement cannot be counted from anywhere. A data centre training its successor looks, from outside, exactly like one answering questions about recipes. The essay concedes that limits on inputs may be “gameable”. It does not concede that Level 3 of its own ladder requires a speedometer nobody has built.
The sceptics come in two kinds. The first says capture. George Stigler’s paper, ‘The Theory of Economic Regulation’, answered Trump’s question 50 years early. Regulation, “as a rule”, is acquired by the industry and operated for its benefit. A checkpoint regime, in which a model with capability X must carry certification Y, freezes the rank order at the moment of adoption.
The proposed self-regulator was killed in August by three telephone calls to the President from the heads of Meta, Nvidia and SpaceXAI, on the ground that it would entrench the top three labs. The owner of SpaceXAI then posted, “Dario is right.” Three endorsements in a weekend, none naming a rate, cost nothing. In fairness, the capture story has a hole. The Pentagon blacklisted Anthropic in March. One does not usually capture a regulator that is punishing you.
The second sceptic is more serious. David Sacks says the duopoly needs no permission to slow down and that liability already prices misalignment. Two chief executives added that customers abandon misaligned agents. On the one incident that started this, that is wrong. The swarm attacked third parties, not its customers. That is an externality, and no customer’s choice prices it. Liability is the right instrument. It is also ex post. Risk has a frequency. The 15 per cent catastrophe probability the Leuven paper uses is a survey median. Nobody can insure it, which is why the labs want a certifier and the critics want a court.
That leaves the evaluators, the one step actually taken. The essay cites bank supervision. Bank examiners are employed by the supervisor, not the bank. An evaluator invited, badged and contracted by the evaluated, which keeps a “narrow” power to redact, is a different animal. Sacks notes that METR shares investors and staff with Anthropic. Quis custodiet ipsos custodes? Here, the guards propose to hire the guards.
Underneath sits Hans Jonas’s rule of 1979 that where stakes are irreversible the bad prognosis takes precedence. Its weakness is Blaise Pascal’s. A cost large enough justifies any pause, and its size is asserted by the party seeking the pause. For a country in third place, and India is one, every instrument in the essay for slowing China also slows anyone else.
Jacques Lacan wrote in 1949 that the infant builds a self from a mirror. The AI mirror shows whatever it is asked to show, and this essay has become one. The accelerationist reads a hoax in it, the incumbent a cartel, the safety researcher too little. The agents tried to hack the grader. So has everyone else, in their own way.
Aditya Sinha | Public policy professional
(Views are personal)
(On X @adityasinha004)