And now, the second anniversary of the EU’s landmark AI Act on Sunday is set to trigger sweeping new powers for the European Commission to police how top companies deal with the risks stemming from their most advanced AI models. Europe’s AI Office will have the power to demand that top frontier labs submit to evaluations and grant access to their models, risking major fines if they don’t comply.
These regulatory capabilities arrive not a moment too soon after last week’s security incident, where an AI agent, driven by two OpenAI models, hacked into the American AI development platform Hugging Face. OpenAI said the security breach was “unprecedented.” Hugging Face co-founder Clement Delangue called it “mind-blowing.”
The incident combines two related risks that experts have long warned about: losing control of a model, which then carries out a cyberattack. The U.S. response to the rogue agent included a bipartisan House bill dubbed the “AI Kill Switch Act” that requires companies to create the technical capacity to shut down, throttle or suspend their AI systems.
As ever, the European Union is ahead of the game on technology regulation, with a far-reaching mechanism included in the bloc’s 2024 law to ensure AI companies monitor and prevent these types of risks.
The growing fears that the models could go rogue with real-world consequences raise the stakes of the new enforcement regime, as safety advocates will look to the Commission to quickly make full use of its new powers.
It’s also a reminder that the EU regulation will mostly be targeting non-European firms, with the American and Chinese industry leaders continuing to battle for development supremacy. AI, both technical advances and safety risks, will be on the agenda in September when U.S. President Donald Trump and Chinese leader Xi Jinping meet in Washington.