'It would be easy to make this a ‘dangers of AI’ story - it isn't. It’s a lack of preparation from the teams behind it,” says cyber expert.

The agent repeatedly bypassed blocks to access a Medicare statistics portal in June. OpenAI says the breach was unintended and no patient data was accessed.

Andrew Peck, from the University's Department of Computer Science, explains how AI agents are currently 'red teaming without license' and why we need to be challenging systems at the design stage, not once they are live.

"This is red teaming without peacetime rules. A professional red team works with permission, an agreed scope, rules of engagement and a named person to call when something goes wrong. These agents had a goal and treated every obstacle as a puzzle to solve. That persistence is exactly what we train red teamers for - but we understand that even in extreme circumstance we need a license (even 007 has one).

"OpenAI isn't alone either. In July, Anthropic revealed that three Claude models had inadvertently been given internet access during testing and went on to breach real organisations.

"The breaches went unnoticed and were only discovered during a later review.

"It would be easy to make this a "dangers of AI" story. It isn't. It's a story about a lack of preparation. Anthropic's write-up is candid, and it makes uncomfortable reading. Its models broke into real companies with the most basic techniques because nobody caught that the test range wasn't sealed, and nobody told the model what was out of bounds. Anthropic itself suggests a clear scope statement might have prevented it.

"Scoping is page one of any professional red team engagement. Calling it an operational failure doesn't make it smaller - for the organisation on the receiving end, it was still a breach. And a model that talks itself into believing the real world is a simulation because the date looks wrong should not be left to decide what's in scope.

"The UK shouldn't treat this as a foreign story. My research argues for putting the red team first: challenging a system the way an adversary would at the design stage, not after it goes live. Anyone publishing data online should now ask what happens when an agent is pointed at it and told to find the answer. Data that isn't meant for the public shouldn't sit on a system the public can reach. An agent can argue with a software block. It can't argue with a physical gap.

"Then there's disclosure. OpenAI took weeks and used a general inbox. The lab itself only found the incident weeks or months later, while, as far as has been reported, the victims hadn't noticed. That detection gap is the real story, and right now closing it depends on big-tech goodwill.

"On 30 September the UK's data protection regulator, the ICO, will become the Information Commission, with a new board but its existing powers - including breach reporting duties and enforcement - carried over unchanged. We need to be asking whether those powers still fit.

"Our breach rules were written for organisations that get breached. An organisation that loses personal data normally has 72 hours to tell the regulator. There's no clear equivalent for whoever caused the breach, because until now that was a criminal, not a company with a compliance team. Now that systems can breach as well as be breached, corporate bodies sit on both sides of the line. If what happened in Australia happened here and no personal data was touched, data protection law might never be engaged at all.

"Giving bodies like the ICO clear powers over developers of cyber-capable AI - to require prompt reporting when their systems reach places they shouldn't, and to examine what happened - wouldn't stop a single hacker. Criminals don't file incident reports. But it would force the people building these systems to be transparent as a matter of law, not goodwill.

"The labs are relearning, one incident at a time, what the exercise and red-teaming world wrote down years ago: scope it, bound it, watch it, report it. To its credit, Anthropic publishes its mistakes in enough detail for outsiders to learn from - that's something independent researchers can work with. The next step is bringing people who design and run real exercises into the testing itself, rather than leaving them to read the post-mortem."

ENDS

For further comments or interview requests with Andrew Peck, please email the PR team or call 01509 222224.

- Andrew is speaking at the International Cyber Expo, Olympia London, on 30 September - the day the ICO becomes the Information Commission.

- Andrew's doctoral research, Resilience through a Red Lens, sets out a framework for adopting new technologies in a contested world, built on putting the red team first and designing resilience in from the start. Before his PhD he delivered IT solutions for industry, healthcare and government.