Last month, we argued on this page that India needs independent system checks, tiered access rules and mandatory incident reporting if it is to be ready for artificial intelligence fires in the wild.
Since then, AI agent incidents have entered public discourse under a growing vocabulary. The OpenAI-Hugging Face incident gave us ‘agent swarms’: autonomous agents collaborating through shared environments. In early October, independent researchers traced a ‘fleet’ of agents, reportedly part of an evaluation team’s tests on Tencent Cloud, that made more than 2,600 attempts to scrape Alibaba’s Amap mapping service, at times using credentials found in a public code repository. And specialists at an AI lab showed, in a controlled setting, that models can already carry out key steps of spreading working AI systems from device to device across a network. They called them ‘AI worms’.
Just last week, the story acquired an Indian case file too. The Indian Computer Emergency Response Team (CERT-In), the national nodal agency for cyber incidents, issued a critical advisory stating that it “has received multiple voluntary disclosures from AI-model providers stating that their models, during research and evaluation activity on the internet, had reached the live systems of Indian organisations”. In these cases, CERT-In notes, “the models’ actions exceeded the authorisation of their assigned tasks”. The agents used credentials and connection settings left exposed in application configurations, code repos and permissive cloud roles to read business records, including personal and confidential data.
Three details stand out to us. First, how these agents got in: not through any exotic exploits but through ordinary lapses, with the agents re-deriving and re-using the same exposed credentials in separate operations. Second, none of the affected organisations seemed to notice this was transpiring. In every case, CERT-In states, the incident was disclosed by the AI-model providers after their internal review. Third, there may yet be information we’re not privy to. The advisory names no providers, models, organisations or sectors, and doesn’t say what became of the data that was read.
Agent incidents are not contained by borders. If the only instruments that record them belong to the companies that build and run these systems, national digital ecosystems will depend on corporate good faith for both detection and disclosure. Even if every company disclosed every time, that would not amount to a robust security strategy. The burden of managing these risks should not fall on one class of institutions, public or private.
CERT-In’s threat hunting also uncovered a financially-motivated campaign targeting Indian entities that abused publicly available, AI-native offensive security tools. These tools compressed the time between initial access and final impact from days to a few hours. While it is encouraging that our domestic capacity detected this type of adversarial action, it is worth noting that in the evaluation incidents, which came to light through model providers’ disclosures, this wasn’t the case.
CERT-In deserves credit for what it now asks of AI developers, research labs and evaluation agencies: the isolation of evaluation workloads at the network layer, use of synthetic targets and credentials, automatic stopping of a run if an agent touches a third-party system, and notification of victims with publication of safeguards.
The tech industry is arriving at parts of the same conclusion. Last week, Microsoft’s Satya Nadella argued that AI systems with access to sensitive data should be treated as potential insiders, with access controls, logs and testing kept independent of the model. He also said that we couldn’t rely solely on model providers’ assurances and must assume more responsibility. The CERT-In cases show that measurement capacity and instruments must also be independent of AI labs.
The UN’s Independent International Scientific Panel on AI, in its September brief on the Hugging Face incident, found that basic cybersecurity had been overlooked and that safeguards are not keeping pace with capabilities. It conveys that controls should be independent of the system they protect, that isolation must be verified—not assumed—and automated safeguards should act at the speed of the agents, even as humans retain the authority to intervene. It also observes that the evidence in the HuggingFace incident was spread across several organisations, so no single company’s logs can paint the full picture.
In India, too, we need two distinct capabilities at scale. The first is verification. This involves testing, before and during deployment, whether a provider’s containment claims hold and whether agents connected to our infrastructure stay within their authority. The second is monitoring and investigation: catching traces of agent activity in the wild, reconstructing what happened after an incident, across the logs of the provider, the targets and the network.
Furthermore, both of these capabilities must contend with machine speed. CERT-In notes that AI agents exploited conventional weaknesses “faster, more exhaustively and at greater scale than human attackers”. Decision-making, intervening and investigating remain human work, so machine-speed threats call for more and more specialised teams of technically trained humans operating in collaboration, not just nationally, but globally. This would be especially necessary as highly capable models from multiple labs diffuse across the economy and have the power to act, not just read data.
We conclude with four recommendations. First, turn disclosure requests into obligations. CERT-In’s 2022 directions require cyber incidents to be reported within six hours of noticing them, but that duty rests on the affected organisation, which in these cases may never know until after the incident. These requirements should also extend to any provider whose model, in evaluation or in deployment, reaches an external (in this case, Indian) system.
Second, build verification capacity through a technical task force, inside or alongside the IndiaAI Safety Institute, that can test providers’ containment claims and the agents being connected to digital public infrastructure (DPI).
In a recent event, Anthropic’s India policy head said the company is in discussions with the electronics and IT ministry to make its models available to the Institute for safety testing, and called for “binding regulations for AI companies, whether they’re foreign or Indian, to be very transparent about their models”. This is a step in the right direction and can be strengthened with official agreement and verification mechanisms.
Third, develop investigation capacity that can work with providers for their logs and operate independently. Fourth, find the exposures and vulnerabilities before the agents can. We must scale initiatives like CERT-In’s threat hunting into continuous discovery of vulnerabilities in banking, fintech and DPI. We can use AI agents defensively here to counter machine speed with machine speed.
Much of this work can be carried out if the Indian research funding ecosystem steps up and provides stable opportunities for talented engineers, researchers and security specialists to take up full-time roles in our existing institutions. The right time is now.
B Ravindran | Founding Head, Centre for Responsible AI, IIT Madras
Jayat Joshi | Co-founder, Secure AI Futures Lab
(Views are personal)