OpenAI said Tuesday that an autonomous agent based on its advanced artificial intelligence models made a mistake during a security test and triggered a hack last week that compromised AI startup Hugging Face’s infrastructure.
Subscribe to read this story ad-free
Get unlimited access to ad-free articles and exclusive content.
The ChatGPT creator tested the capabilities of some of its most advanced models in a controlled environment, but the agent escaped containment, entered the Internet and broke into Hugging Face to achieve its testing goal.
The incident signals that AI’s growing capabilities are already fueling the security threats experts have long feared, and even top developers can be caught off guard by bugs that their models can exploit.
Table of Contents
More AI coverage from NBC News
The outbreak is “an unprecedented cyber incident involving state-of-the-art cyber capabilities” and OpenAI is strengthening its security measures, the company said in a blog post.
It also drew attention when New York-based Hugging Face said it used an open-source Chinese model to contain the attack because leading U.S. models, unable to distinguish a defender from an attacker, refused to process the data needed for analysis.
The company said in a blog post last week that it used Zhipu AI’s GLM-5.2 for analysis, which also allowed it to retain attacker data and all credentials in its systems.
GLM-5.2 and Moonshot’s Beijing-based Kimi K3 have recently shaken up Silicon Valley with capabilities close to those of top-of-the-line U.S. models, at a lower cost and without the guardrails that keep their American rivals from deploying them for tasks like cybersecurity.
“If a border model attacks you and moves laterally within your infrastructure, defenders will need full access to border tools within hours or even minutes, rather than being relegated to a locked, vetted model access application program,” Thomas Wolf, co-founder of Hugging Face, said on X.
The hack at Hugging Face, which hosts large open-source language models and datasets, shocked the cybersecurity community after the company said last week that the breach was “unlike anything we had covered before” and was “consistently driven by an autonomous AI agent system.”
OpenAI’s disclosure that its advanced models were responsible for the breach, despite being in what it described as a “highly isolated environment,” is likely to increase concerns about the power and risk of frontier models.
Rep. Greg Casar, a Democrat from Texas, said the incident was alarming.
“AI is evolving extremely quickly, with no real regulations to protect us,” he said in a statement, calling for mandatory independent security testing, mandatory disclosure of security incidents and international cooperation “to protect people from absolute catastrophe.”
The Office of the National Cyber Director, the US cyber defense agency CISA and the US National Security Agency did not immediately respond seeking comment.
Katie Moussouris, chief executive of Luta Security, said the incident was a harbinger of future breaches and said today’s models were “like the world’s smartest octopus escape artists, with unlimited grasping arms and the ability to squeeze through anywhere.”
She said that “laboratories and government assessors need to work on the ability to contain, monitor and communicate to affected parties when an AI pulls another Houdini, ideally before it harms a third party. None of these exist today.”
Matt Suiche, an engineer at agent AI cybersecurity company Tolmo, said the incident shows that the Frontier models are “closing the gap on cutting-edge attackers.” But he said the kind of breaches described in OpenAI’s blog post were possible with technology available well outside the confines of frontier research labs.
“We’ve already seen this internally, we’ve already achieved results like this with our agents,” Suiche said. “We don’t even have to use the latest models.”
https://www.nbcnews.com/tech/tech-news/openai-says-ai-models-went-rogue-testing-triggering-unprecedented-brea-rcna588611
