An AI model hacked another company. What does that actually mean

By Sian Wilkerson

OpenAI recently announced that some of its experimental artificial intelligence models escaped its network constraints and used a previously unknown exploit to access another company's production systems.

In a statement this week, OpenAI called the incident unprecedented and said it would share preliminary findings "to help defenders understand what happened and to help calibrate on what models are now capable of."

But according to cybersecurity expert Christopher Whyte, Ph.D., this is not quite the crossing of the Rubicon that some are professing.

"The question of significance here somewhat comes down to whether or not an AI model actually hacked a company on its own," said Whyte, an associate professor of homeland security and emergency preparedness at Virginia Commonwealth University's L. Douglas Wilder School of Government and Public Affairs. "In one important sense: Yes, it did. In another sense: It just did things we've observed AI systems doing for many years."

VCU News caught up with Whyte, whose research examines cybersecurity, artificial intelligence and emerging technologies in national security, to understand what these developments mean for AI - and what's next.

So what happened, exactly? And is it really unprecedented?

In essence, OpenAI was testing advanced models on a cybersecurity benchmark. The test was being conducted in an environment where model guardrails had deliberately been reduced, which already makes this a special circumstance.

The models in question, one of which was pre-release, were given a task to accomplish. In pursuing that task, they found a way around the restrictions on their environment, reached the open internet and ultimately compromised systems belonging to AI company Hugging Face, which has since talked about the intrusion as being conducted end to end by an autonomous AI agent system.

Obviously, the investigation is still ongoing, so some technical details might change. But what happened wasn't that AI woke up one morning and decided that it wanted to hack somebody. The human testers supplied objectives, the computational resources, the tools and the environment. What the AI then did was determine that gaining access to information held by Hugging Face could help it achieve the objective, and it pursued that path without being instructed to do so..

You're saying this is in line with what we've previously seen from AI.

It's analogous to the standard example of AI that's asked how to most efficiently and economically ford a river, prompting it to suggest building a tall building and then just toppling it over the water to make a bridge of sorts. It's not really malicious, just anomalous from the perspective of human problem-solving.

In a nutshell, that's what happened here, too. That's the "No, it wasn't hacking on its own" case.

And on the "Yes, it was" side of things?

I think what is striking is how much of what happened between the objective and the outcome was determined by the system itself. That's critical because we clearly don't need AI systems to develop malicious intentions for them to create serious security problems: We only need systems capable enough to find unexpected ways of accomplishing the goals we give them.

What does this mean for the capabilities of AI?

We have obviously known for some time that advanced AI models are getting better at individual cybersecurity tasks, particularly routinizable tasks like finding vulnerabilities, writing code, analyzing systems and assisting with exploitation.

I do think this incident points toward something more consequential, in that those capabilities can increasingly be assembled into longer chains of activity in which an AI system encounters an obstacle, changes its approach and continues pursuing an objective without a human specifying every intermediate step. That is an important threshold for cybersecurity, because a great deal of sophisticated cyber activity has traditionally depended on scarce human expertise and considerable amounts of human attention and creativity.

Increasingly, autonomous systems can potentially reduce those requirements, which erodes a traditional barrier to entry and sustained operation for malicious actors that has shaped our defensive, deterrent and response postures for decades.

The idea of autonomous AI strikes fear into a lot of hearts. But there's a difference between autonomy and intent, right?

I think it's worth asking/answering the question of whether AI systems are becoming more autonomous vs. something else happening here. The short answer is: Yes, they are becoming more autonomous in a practical sense. But there's danger that comes in when we confuse autonomy with intention or consciousness.

Autonomy in this vein means that I can give a system an objective without specifying every action required to achieve it. The system can break the problem into smaller problems, use tools, observe what happens, adapt when something fails and continue working toward the objective. The longer and more complicated that chain becomes, the less directly a human operator determines what happens at every stage. That gets us a range of tricky and sticky governance problems even if the AI has no inherent desires whatsoever.

That may be the most useful way to think about this incident. Humans stay responsible for deciding what these systems are asked to do, how they are initially deployed and with what resources. But increasingly capable systems introduce a growing gap between specifying an objective and predicting exactly how that objective will be pursued.

What's the takeaway here?

One lesson for AI companies and other organizations is the most likely failure around these frontier systems is socio-technical failure. AI safety cannot depend entirely on getting the model itself to behave properly - we also have to design the environment around these systems assuming that sometimes they will behave in ways we did not anticipate.

That's obviously a pretty familiar cybersecurity problem, which makes this OpenAI failure a bit more damning than if the incident were truly without precedent. Organizations already use principles like least privilege, network segmentation, etc. because we assume that individual security controls will sometimes fail. These highly capable AI agents should have been treated in much the same way.

Going forward, models like these should only have access to the systems and information necessary for their tasks, and their ability to communicate outside controlled environments should be carefully constrained. Organizations need good visibility into what they are doing, so some version of forcing systems to explain themselves is critical.