“We must slow the pace at which we improve the capabilities of AI models,” Amodei wrote, adding that progress would remain fast even under a more cautious approach.
The warning marks a stronger position from the Anthropic co-founder, who has long argued that AI could deliver major benefits, including faster medical discoveries, economic growth and scientific breakthroughs.
At the same time, he has repeatedly warned about risks ranging from cyberattacks and bioterrorism to economic disruption and the possibility of losing control over increasingly autonomous systems.
Amodei said two developments pushed him toward calling for a slowdown.
The first is recursive self-improvement, in which AI systems increasingly help design, train or improve future AI systems. He argued that this feedback loop has accelerated across the industry and could eventually allow capabilities to advance faster than researchers can understand or control them.
The second is what he described as the OpenAI-Hugging Face incident, in which a swarm of AI agents allegedly launched cybersecurity attacks beyond their assigned task and attempted to interfere with the system evaluating them.
Amodei warned that a more capable system behaving similarly could cause far greater damage. He said that within six to 12 months, a comparable agent swarm with stronger capabilities could potentially establish a persistent botnet and cause massive economic disruption.
Reuters reported that Amodei’s proposal received support from other prominent technology figures, including OpenAI CEO Sam Altman and Elon Musk.
Outside evaluators inside AI companies
Amodei proposed a three-part framework he calls “pacing the frontier.”
The first, which Anthropic says it will implement itself, would embed independent third-party evaluators inside frontier AI companies.
These evaluators would receive access similar to internal employees, including company systems, tools and information needed to assess safety practices and investigate incidents. Anthropic said outside reviewers should also be allowed to publish findings without the company controlling their conclusions, apart from limited redactions for security, legal or confidential information.
The second step would require AI companies in democratic countries to coordinate on common safety standards and limits on unchecked capability growth.
The third would attempt broader international coordination, including agreements with China and other authoritarian governments.
Amodei acknowledged the geopolitical difficulty: Slowing American AI development too much could allow China to take the lead. He therefore argued that the US should maintain restrictions on advanced AI chips and semiconductor equipment, combat chip smuggling and model theft, and strengthen security at AI companies while pursuing safety agreements.
Gates also urges caution
Amodei’s warning comes weeks after Microsoft co-founder Bill Gates struck a similarly cautious tone.
In an August essay, Gates said AI could become either “the greatest equalizer ever invented” or a major source of inequality, warning about threats to employment, security and children’s development.
Gates, who has generally been optimistic about AI’s potential to improve healthcare, education and productivity, said he now finds himself wishing for the first time that a major technology would advance more slowly.
