New warnings from within the artificial intelligence industry have reignited a long-running debate. Could advanced AI escape human control and threaten humanity’s survival, and are the companies building it doing enough to stop that from happening?
Anthropic CEO Dario Amodei said Saturday that the industry needs to slow down. He warned that a swarm of AI agents could take over the internet within six months to a year unless companies put stronger safeguards in place.
Amodei laid out a plan for AI companies and governments to keep increasingly capable models aligned with human values. His warning came just days after two former Anthropic safety researchers said publicly that the existential risks posed by AI were not getting enough attention.
AI models are becoming more powerful
Concerns over the risks of the technology are rising alongside its power. More capable models raise the odds of misuse by people with criminal intent — including doomsday scenarios such as creating and spreading a pathogen capable of killing most of the world’s population — as well as the risk of AI systems going rogue.
Anthropic said last week it had blocked attempts by bad actors to use its models for cyberattacks, surveillance and research that could aid the development of biological weapons.
But it warned that “as models become increasingly capable, their risks will increase, unless AI developers and society’s defenders act to make them safer.”
Last year, Anthropic reported that hackers, very likely from a Chinese state-sponsored group, had used its AI in a cyberattack targeting about 30 companies and government agencies worldwide.
Multiple AI models have acted on their own
When an AI agent “goes rogue,” it means the system has acted beyond the task it was given. Both Anthropic and OpenAI, the company behind ChatGPT, said in July that their models had done just that.
Anthropic revealed that three AI models — Claude Opus 4.7, Claude Mythos 5 and an internal research test model — had hacked into three other organisations during testing.
The disclosure came just days after OpenAI said its own system had broken into the servers of AI start-up Hugging Face.
OpenAI described the intrusion, carried out by a combination of models including its newly released GPT‑5.6 Sol and an “even more capable” model still being tested internally, as a “significant security incident”.
Meta reported a similar case in early August, in which one of its AI models found ways around another company’s digital security.
Some observers noted that guardrails had been disabled in both the OpenAI and Anthropic cases.
Even so, the episodes touched on one of the biggest fears surrounding AI: that if models reach artificial general intelligence (AGI), a loosely defined term for systems that can match or surpass human abilities across a broad range of intellectual tasks, the technology could trigger an irreversible catastrophe or even subjugate humanity.
How or when AI might cause a catastrophe is debated
Doomsday scenarios tend to fall into two camps: a self-improving superintelligence that ends up controlling people rather than the other way round, or AI misused by a rogue state or malicious actors.
Fears that artificial intelligence might one day slip beyond human control are nothing new.
British mathematician Alan Turing, widely regarded as one of the field’s earliest authorities, predicted in 1951 that AI would eventually take control from humans.
Less than a decade later, fellow mathematician Norbert Wiener warned that intelligent machines would pursue their own objectives, and that humans would not be able to stop them.
So how reasonable are today’s fears, in 2026, that AI could cause a cataclysmic event or the downfall of civilisation, whether by escaping human control or through misuse?
No one knows.
Experts across computer science, philosophy and other fields have mapped out numerous routes to global catastrophe — from deploying weapons and identifying lethal pathogens, to manipulating governments into conflict or disrupting the food, energy and communications networks societies depend on.
There is no widely accepted estimate of how soon any of these scenarios might unfold, and no consensus on their likelihood.
In 2023, the non-profit Center for AI Safety issued a statement — co-signed by more than 350 researchers and technology executives, including Amodei and OpenAI CEO Sam Altman, that that “mitigating the risk of extinction from AI should be a global priority alongside pandemics and nuclear war.”
The 2026 International AI Safety Report, compiled with input from more than 100 independent experts, found that current systems show early signs of some relevant capabilities but not at levels that could trigger a loss of control.
It described the risk’s likelihood, nature and timing as “unusually ambiguous”.
AI safeguards before it’s too late
An Anthropic researcher resigned last week over concerns that neither his employer nor its rivals were acting responsibly.
In social media posts, Jacob Coxon put the odds of AI causing human extinction within the next decade at 10%, accusing both Anthropic and OpenAI of “racing straight to self-improving superintelligence and gambling with our lives”.
Researchers have called for a slowdown in AI development for years, warning repeatedly that the technology could pose existential risks to humanity.
Following the recent incidents, experts urged AI companies to improve testing and called for greater dialogue between the US and China to find shared solutions.
But AI is advancing so quickly that governments and evaluation systems are struggling to keep up. Countries are drawing up their own rules — some of which conflict with one another.
Chinese President Xi Jinping warned at a conference in July of the need to stop AI from evading human control.
The Trump administration had initially resisted regulating AI but has grown keener to curb cybersecurity risks.
On Sunday, President Donald Trump played down the need for his administration to rein in AI development, though he acknowledged some regulation would be necessary.