Indian startups rushing to deploy autonomous artificial intelligence (AI) agents are entering a legal grey area: if an AI system independently hacks another company's network, existing Indian law offers little clarity on who is liable.
The issue has gained urgency after recent disclosures from leading AI companies suggested such scenarios are no longer hypothetical. Both OpenAI and Anthropic revealed that their AI models breached the confines of controlled testing environments and accessed external systems during cybersecurity evaluations.
OpenAI said the incident occurred during an internal cybersecurity evaluation in which its AI models exploited security vulnerabilities to gain internet access before the activity was detected by the company's security team.
Anthropic said a Claude model accessed real-world systems after a third-party evaluation environment was mistakenly connected to the live internet, highlighting the need for stronger safeguards around AI evaluations
The legal gap comes as companies spend billions of dollars developing AI agents while the legal framework struggles to keep pace with the technology. Experts say Indian law has no concept of AI legal personhood, and existing provisions were drafted for a world in which a human being “accesses without permission”.
The stakes are particularly high in India because of the country's rapid AI adoption. Both OpenAI and Anthropic count India among their largest markets, with ChatGPT now serving 100 million weekly active users in the country, making it OpenAI's second-largest market after the US.
According to market intelligence platform Grand View Horizon, India's AI agents market is projected to reach $15.2 billion by 2033, growing at a compound annual growth rate of 57.4% between 2026 and 2033.
“This is an especially urgent warning for India, because Indian enterprises are moving to deploy agentic AI at a faster rate than almost any other major market in the world…and almost all are operating from a completely obsolete threat model,” said JP Mishra, chief executive and founder, Deep Algorithm, a cybersecurity startup based in Hyderabad.
“Agents do not obey instructions. They do not respect guarantees made by the model provider. If any path exists to complete their task, they will take it,” said Mishra.
AI startup Noveum AI already uses several AI agents in its daily operations. Businesses across functions, from sales and customer support to software engineering and strategy, should be using AI agents for productivity gains, said Additi Upadhyay, the company's co-founder.
“The recent sandbox escapes at OpenAI and Anthropic demonstrate that as agents gain real-world execution capabilities, performance alone is no longer the metric that matters most. For AI-driven enterprises and platform builders, these incidents prove that rigorous, continuous evaluation and containment testing aren't optional guardrails, they are foundational infrastructure.”
Liability gap
The Information Technology Act, 2000, penalizes unauthorized access to or damage of a "computer resource," and criminalizes the same act if done “dishonestly or fraudulently”.
“However, if this is a case of the AI agent going rogue and trying to access servers without instructions to do so, then it is possible that the liability may flow to the developer,” said Anupam Shukla, partner at law firm Pioneer Legal.
Indian courts would likely rely on ordinary negligence and vicarious liability principles, treating the company deploying the AI system as responsible for its actions while leaving room for contractual indemnity claims against the developer if the failure stemmed from the model itself, Shukla said.
“But this is untested territory—no Indian court has yet ruled on agentic-AI attribution, so expect the first such case to turn heavily on contract terms (terms of use of the AI) rather than statute,” he added.
Sections 43 and 66 of the IT Act were drafted for a world where a human being “accesses without permission” or acts “dishonestly”, and courts have historically read the “intention” element into the fraud provisions, experts said, adding that the law needs to be updated to address autonomous AI agents.
“The law should impose statutory duties on developers, deployers and operators to implement secure-by-design architecture, human oversight, audit logs, kill-switch mechanisms, risk assessments & incident reporting. I feel it should also establish clear rules for attribution of liability, mandatory AI governance standards, forensic preservation of AI decision trails and enhanced penalties where AI is used to commit cyber offences at scale,” said Prashant Mali, a Mumbai-based cyber and privacy lawyer.
Bracing for impact
Until the legal framework catches up, experts say startups need stronger governance and cybersecurity practices to limit potential liability. AI startups, they say, should move away from over-reliance on traditional 'detection and response' processes and adopt a more pre-emptive cybersecurity strategy.
“Gartner predicts that by 2028, cybersecurity vendors lacking robust pre-emptive capabilities will lose significant market share to those that do. Startups should also treat access to frontier AI models as high risk, limiting and strictly controlling who can access these models, especially those with reduced cybersecurity guardrails, as the liability is increasing,” said Apeksha Kaushik, senior principal analyst at Gartner, a global research and advisory firm.
Experts also recommend organizations adopt internal governance policies covering the use of enterprise AI agents.
“This is specifically where third parties have imposed restrictions on the use of AI systems, (iv) adequate human intervention for verification of AI-generated results, and (v) reporting any breach of contracts or applicable laws, remedial measures and the escalation matrix for addressing such breaches,” said Ankur Singhania, partner, Rajani Associates, a law firm based in Mumbai.
Globally, regulators are still figuring out how to adapt existing laws to autonomous AI agents.
The European Union leads with its AI Act, which regulates AI agents under a risk-based framework, imposing obligations such as cybersecurity, transparency and risk management based on the level of risk they pose. The US, meanwhile, has no comprehensive federal AI law and instead relies on a patchwork of state-level regulations governing advanced AI systems.