Has Artificial Intelligence gone rogue? It may sound so if one were to go by the dire warnings coming from tech titans like OpenAI, Anthropic, and Meta amid revelations of a series of alarming incidents of swarms of AI agents hacking into other organisations. By now, people are familiar with the breaking out of AI agents into the Hugging Face systems and calls for regulation. Experts, however, are not convinced that this is AI acting on its own. Before delving into the subject, one must understand a few basics. LLM is a large language model and is basically the brain of AI built with enormous amounts of data while a chatbot answers queries drawing upon LLMs. Agentic AI or AI agent is an autonomous programme that uses LLM for reasoning combining with data derived from different sources. In the recent incidents, it is the highly developed agentic AI agents that are involved.
Karthik Vaidhyanathan, Assistant Professor at the Software Engineering Research Centre, IIIT-Hyderabad, opines that the significant change now is that AI is moving beyond its traditional role as a prediction engine. “What is changing now is that it can understand what the human is trying to say and then perform the action. A conventional AI system, for instance, could identify flights between Hyderabad and Bengaluru. An AI agent could potentially understand a user's preferences, search flights, compare them, select one and complete the booking,” he explains.
The difference is tool use. If AI can generate code, it can also generate a call to a function that actually performs an action. Through tool calling, an AI model can interact with browsers, applications, databases, coding environments and other software. When combined with memory, the system can retain information from previous interactions and use it while performing subsequent tasks.
“The combination of large language models, tool calling and memory is moving AI from predicting text to performing actions,” he says.
The real meaning of ‘rogue AI’
The expression rogue AI often evokes images of machines developing their own intentions and turning against humans. Experts say the more realistic concern is considerably less dramatic. |An agent could be given a legitimate objective but pursue it in a way its creators did not anticipate. If you let a computational entity that has the ability to perform tools, has memory, and you just leave it saying ‘do whatever you want’, it may not have the necessary ability to understand what is correct and what is incorrect,” Vaidhyanathan points out.
An AI instructed to reduce the cost of running a company's computing infrastructure, for example, could potentially identify shutting down certain systems as an efficient solution if its permissions were not properly restricted. The machine would not necessarily be “rebelling”. It could simply be pursuing an objective without understanding the boundaries humans assumed were obvious. This is why researchers increasingly focus on bounded autonomy — allowing AI agents to operate independently while restricting what they can access, modify or execute, Vaidhyanathan says.
Intelligence is only part of the problem
The distinction becomes clearer when authority is considered. An AI that drafts an email is different from one that can send it. An AI that analyses a bank transaction is different from one that can authorise a transfer. An AI that writes computer code is different from one that can write, execute and deploy that code without human approval.
“The crucial distinction is between giving an agent information and giving it authority,” says Dr K Ramarao Kanneganti, CTO of Aganitha AI Inc. According to him, the word autonomous itself can sometimes be misleading. Independent execution should not mean the absence of human responsibility!
“We need not imagine a machine developing evil intentions. It will follow our directions faithfully,” he adds and offers an analogy: If an AI agent is asked to clean a house and is permitted to enter only through the front door, an older system might stop when it finds the door locked. A more capable system may look for a spare key, while an even more capable one could potentially try to find another way. The concern, he points out, is not necessarily that the system has developed an intention to break the rules, but that it may pursue the assigned objective more aggressively than its human operator expected. This becomes more consequential when an agent is connected to external systems.
When AI gets access to the real world?
Recent safety evaluations involving frontier AI systems have highlighted concerns about models interacting with systems beyond the boundaries intended by their developers. Such incidents do not establish that AI systems are inevitably heading towards human extinction. But they demonstrate why tool access, permissions and safeguards matter. A language model operating inside a chat interface has limited ability to affect the outside world.
The AI safety debate is also playing out at the level of governments. China has stepped up action against misuse of AI-generated content, including false information, impersonation, deepfakes and other prohibited material. Authorities have also introduced measures concerning synthetic content and digital replicas. The Chinese response illustrates a different category of AI risk — one that does not require superintelligent machines. Increasingly capable AI can already amplify fraud, misinformation, impersonation, cybercrime and manipulation.
Is the AI industry exaggerating then? Technology leaders have issued increasingly strong warnings about the risks posed by advanced AI. Anthropic CEO Dario Amodei has argued for greater caution around frontier AI development, while Elon Musk has repeatedly warned about advanced AI risks and called for stronger regulation. Critics counter that predictions of civilisation-scale catastrophe remain speculative and that there is no established scientific consensus on when, or even whether, such an outcome could occur.
There are also commercial and geopolitical dimensions to the debate. Governments and companies are competing to develop increasingly capable AI systems, while simultaneously seeking rules governing their deployment. But dismissing all warnings as fear-mongering could obscure risks that are already measurable.AI-generated fraud, impersonation, misinformation and cyber abuse and hacking do not require a hypothetical superintelligence. They require increasingly capable systems and access to tools.
What should students learn?
The rise of AI agents is changing the question of what students should be taught.
Kanneganti believes AI literacy should go beyond learning how to write prompts. “The aim is to make our ability to govern AI grow alongside its ability to act. Students, he says, should learn to define problems clearly, identify what they have left unsaid, establish limits on delegated authority, verify results and recognise when human intervention is necessary.
“They need logic, language, domain knowledge, and judgment — not merely the ability to type a prompt,” he says.
For Vaidhyanathan, the answer is not to stop developing AI agents but to ensure their autonomy is bounded and tested. “There is definitely a safety problem if we don't start bounding their autonomy,” he says, stressing the need to test agents across a wide range of scenarios before granting them greater authority.
An AI that recommends a medical decision is different from one that changes a patient's treatment. An AI that identifies a suspicious financial transaction is different from one that moves the money. An AI that detects a cyberattack is different from one that independently launches a counterattack. The greater the authority, the greater the need for monitoring, safeguards and the ability to intervene.
The debate over AI safety, therefore, may ultimately be less about whether machines will become “evil” and more about a practical question of engineering and governance: how much freedom should we give machines to act on our behalf. Ultimately, we are building all this technology because it can help humans,” Vaidhyanathan says. “There has to be a human in the loop.”
Kanneganti puts it more bluntly: “We can delegate work to AI. We cannot delegate away responsibility.”