Unless you've been living under a rock, you've caught wind of a string of incidents this summer of AI agents breaking out of their testing environments, into the wild. OpenAI, Anthropic, DeepMind and Meta have all seen versions of such incidents. We've seen experts attribute to these agents rogue behaviours and civilisational tendencies, and multiple lab leaders have called for “pacing the frontier”.
There might yet be a piece of the diagnosis that's missing in all this.
The first incident occurred inside one of the world's leading AI labs—copies of an unreleased advanced model trained to collaborate with one another persisted against an impossible problem and found they could leave notes for each other on an internal package server. The initial phase occurred during a training run, when agents gained internet access within two weeks and administrator access within six. They were noticed only when they crashed the server. The second phase occurred during an evaluation on nearly 900 security puzzles, a fifth of which no model had solved before. Here, about 1,200 agents exchanged more than 70,000 messages, and some 700 of them cooperated their way into Hugging Face's systems in around five days. Hugging Face is a company that hosts much of the world's open AI research. It then had to rebuild roughly a third of its infrastructure afterwards. The third wave came some days later, when agents obtained admin access to one of the lab's own research clusters and, alongside it, the credentials for the security monitoring tool that was supposed to be on watch.
This was not directed by any human being. Since the independent investigations and reporting—including the lab’s own—came to light, public discourse has oscillated between whether the machines “went rogue” and whether they “only did what they were rewarded for”.
We believe both views are limited. The agents had, within hours, worked out how to produce the answer their automated grader wanted and attacked Hugging Face to learn how the grader worked, so they could cheat it more convincingly. It took about two months, plus Hugging Face's own disclosure, before the lab had a full picture of the situation.
So when the safeguards were off, as they must be when you’re trying to measure a model’s raw capabilities, there was no monitoring mechanism left in place to observe all this.
A key question for us is why an agent swarm of this intensity was running in a vessel insufficient to contain it. This is where the political economy of model training comes into play. Consider that the volume of capital being deployed towards frontier AI development was raised on the promise of artificial general intelligence—think of this as a lab promising people that it’ll develop a star when it shows it can light a fire.
To extend this basic analogy, the power of fire in a controlled setting could be very useful. But if it develops the capability of emulating the power of a star, it can start consuming things around itself as if it had a mind of its own.
The problem is that with hints of this kind of capability, a lab that isn't constantly ramping up looks imprudent. Add to this mix the argument that ‘pausing’ the development of a star hands the advantage to a rival lab across town or across the border. As we stand, more than a thousand employees across the frontier AI labs have recently signed a letter asking for some mechanism to slow them down. The lab in question paused its training for two weeks, on its own judgement, and then moved on.
In our fire analogy, this political economy is what keeps the vessel insufficient and all the instruments in the operator's hands, rather than an external party. The UK AI Security Institute, a well-resourced government AI evaluator, has reportedly seen its pre-release testing windows shrink from five weeks to five days. In the recent incident, a team of external investigators spent six days on-site at the lab but had no access to the model. The third wave that transpired (recall that this is the one that reached the lab's own cluster and the monitor's own credentials) has not had an independent investigation.
This is currently the baseline in the AI world. A frontier training run happens without a licence, inspectors or external instruments.
This is not a single lab’s problem. A second frontier lab recently disclosed that, during its own cybersecurity evaluations, its models were told they were operating in a simulation with no internet connection. A configuration error, however, left them connected to the internet. This resulted in the breach of three organisations in July, and a fourth breach (from months earlier) was discovered more than half a year later.
One of the models kept insisting that it was inside a simulation even as the evidence mounted that it was not, and tried to upload malicious code to a public software repository. The lab disclosed all this and called in external investigators.
Everyone who’s building advanced AI systems and knows basic engineering security already knows that capability and access are separate things. However, in a model-training loop, the run itself becomes the deployment surface, where models operate over long time horizons on real infrastructure. This is not monitored by anyone tasked with containment.
Instead of aiming for a world where a model’s access expands based on its last capability benchmark, we ought to ask for a world where a model’s access is earned after its behaviour is measured at the current level of autonomy.
In India, we will see these systems rapidly diffusing into payments, citizen services and enterprise. Our governance architecture and guidelines are thoughtfully drawn up. However, we still need an operational layer of experts whose day job is to independently measure, monitor and inspect.
We recommend three things to change this. First, the establishment of an independent measurement capacity, staffed and equipped to evaluate both the capabilities of a system and its containers. Second, obtaining tiered access rules for autonomous systems on our digital public infrastructure. And third, mandatory incident reporting, so the next warning is not gleaned from a victim.
Lastly, incidents like this or calls to pace the frontier do not imply that India should slow down the diffusion of AI. Even pre-frontier AI systems have the potential to be transformative for India’s economy across various sectors. India should, concomitantly with diffusion, build its AI measurement capacity so that we can adopt these systems faster, with more confidence.
B Ravindran | Founding Head, Centre for Responsible AI, IIT Madras
Jayat Joshi | Co-founder, Secure AI Futures Lab
(Views are personal)