Artificial intelligence company Anthropic has called for a coordinated global pause on the development of the most powerful AI systems, warning that rapid advances could leave humans unable to control future generations of the technology.
The San Francisco-based company said the capabilities of leading AI models are improving at an accelerating pace and could eventually reach a stage known as “recursive self-improvement”, where systems can develop more advanced successors without human intervention.
In a paper published on Thursday, Anthropic said the world should be prepared to slow or temporarily halt frontier AI development if risks become too great. The company argued that any pause would need to be coordinated among major AI developers and supported by mechanisms to verify compliance.
“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development,” Anthropic said.
The company said AI systems’ ability to perform tasks autonomously has been doubling roughly every four months. It warned that the technology is moving towards a point where it may be able to improve itself without human help.
“If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important,” Anthropic said.
Anthropic researchers Jack Clark and Marina Favaro wrote: “We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for.”
The company said a temporary slowdown would give governments, researchers and society more time to develop safeguards, oversight systems and alignment research to manage increasingly capable AI systems.
Calls for a pause in advanced AI development are not new. Similar proposals were made in 2023 by a group of technology leaders and researchers, but efforts to establish international rules have made limited progress.
Anthropic said it plans to work with policymakers and experts to examine how a coordinated pause could be implemented if future AI systems begin to pose risks beyond existing controls. Other leading AI developers had not publicly responded to the proposal at the time of publication.