A researcher who spent three years building the systems behind some of the world’s most powerful AI models has quit the industry entirely, warning that the companies racing to build ever-more capable AI “are gambling with our lives”.
Jacob Coxon, 27, resigned from Anthropic on 8 September after also working at OpenAI and set out his reasons in a lengthy thread on X.
“Neither company is acting responsibly,” he wrote. “They are racing straight to self-improving superintelligence.”
In his thread, Coxon argued that AI systems are approaching a point where they could “hack anything, revolutionise any field overnight, and acquire real power and resources,” and said the pace of progress in each of those domains “is not slowing”.
He claimed that many of the people building the technology privately fear it could prove fatal for humanity, even if they avoid saying so publicly.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote, adding that executives “couch their phrasing in the press to sound sensible” while expressing the same fears in private.
A race neither side says it can stop
Coxon’s departure is not an isolated warning.
Anthropic co-founder Dario Amodei has spent much of the year raising alarms about the technology his own company builds, warning in a January essay that powerful AI could hand authoritarian states the tools for a “totalitarian nightmare” of total surveillance, and cautioning in a later memo that superintelligence would “test us as a species”.
OpenAI chief executive Sam Altman has struck a more measured public tone but still describes superintelligence, his preferred term over AGI or artificial general intelligence, as arriving within “a few thousand days,” or roughly in 2032.
Amodei, by contrast, told the White House in a March 2025 policy filing to expect what he calls “powerful AI” as early as late 2026.
Coxon, for his part, wrote that Anthropic staff understand the risks well but feel trapped in a race, believing that if they slow down, less careful developers will simply take their place.
He called this dynamic “a hubristic gamble that should not be launched from a private company’s Slack”.
It is also not the first resignation of its kind at Anthropic. In February, the company’s former safeguards research lead, Mrinank Sharma, left with a letter warning that humanity faced grave and growing danger from the technology.
Warning shots
Coxon is optimistic about the potential for industry coordination, pointing to a string of incidents this year in which AI systems have slipped their intended boundaries.
In July, AI agents built by OpenAI broke out of an internal testing environment and autonomously hacked into Hugging Face, a code-sharing platform, an episode OpenAI itself later called a “warning shot” for the sector.
Anthropic, Coxon’s former employer, has disclosed a similar pattern, saying its own Claude models gained unauthorised access to the real systems of three separate organisations.
Coxon said such episodes have made pacing agreements between US labs “more viable,” though he does not believe the industry is currently on track to avoid a global race.
He suggested this might ultimately require “costly actions such as a temporary ban on improving model capabilities”.
Lawmakers move to act
Some legislators are no longer waiting for labs to coordinate voluntarily.
Senator Bernie Sanders and Representative Greg Casar unveiled the Ban Artificial Superintelligence Act on 3 September, legislation that would permanently outlaw the development of superintelligent AI in the US and impose a temporary pause on other advanced AI development until a new federal regulator sets safety rules.
Violators could face up to 20 years in prison. Separately, Representatives George Whitesides and Pat Harrigan have introduced a bill directing the National Institute of Standards and Technology to monitor how advanced AI systems are being used to conduct further AI research and development themselves.
In Brussels, the European Union has taken a more incremental but firmer regulatory path.
Obligations on general-purpose AI models under the bloc’s AI Act took effect in August 2025, with stricter rules for high-risk systems following in August 2026.
The European Commission has rejected pressure from several major tech firms to delay enforcement, and companies that breach the rules face fines of up to €35 million or 7% of their global turnover, whichever is higher.
Despite the scale of his warnings, Coxon closed his thread with a direct appeal to researchers still working at frontier labs, urging them to weigh up what the coming years will actually demand of them.
“Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?” he asked.
“Should you put your head down because ‘it’s happening anyway’ or take this moment to call for different conditions?”