Listen to this article
Jump to a section
In this article
Executive Summary
An AI agent goes off script when it does something its owner did not intend. That happens in three ways. Someone else writes the script, by planting instructions in a page, email or tool the agent reads. An attacker runs the agent, and points its speed at your website. Or the agent misreads its own task and acts on the mistake. Websites sit in the middle of all three. They are where attacking agents probe for accounts and records, and they are where hidden instructions wait for the agents that browse them. The evidence is no longer theoretical. In 2025 Anthropic disrupted an espionage campaign in which AI did 80 to 90% of the tactical work against about thirty targets, including financial institutions and government agencies. Researchers showed a hidden comment on a web page turning an AI browser into a tool for stealing its user’s login code. The UK’s National Cyber Security Centre and OpenAI both say prompt injection may never be fully solved. For banks and government agencies, whose portals hold money and citizen records, this changes who is on the other end of every session and what their own agents can be talked into doing.
53%
of web traffic was automated in 2025, and 40% was malicious bots, Thales/Imperva Bad Bot Report 2026
46%
of account takeover attacks in 2025 targeted financial services, the most of any sector, same report
80-90%
of the tactical work in the GTG-1002 espionage campaign was done by AI, with people at four to six decision points, Anthropic, November 2025
11.2%
of hidden-instruction attacks still worked on a browsing agent after its vendor’s defences, down from 23.6%, Anthropic, August 2025
Core conclusions
- Your website has two new kinds of visitor: agents attacking it at machine speed, and legitimate agents acting for customers. You need to tell them apart and decide what each may do.
- Any agent that reads a web page, email or document can be steered by whoever wrote it. Treat everything an agent reads as untrusted input, including your own suppliers’ pages.
- Because prompt injection may never be fully fixed, the controls that matter limit what an agent can do after it is fooled. Parts 2 and 3 of this series cover detecting that and bounding it.
The argument, in ten slides
Save it, or send it to whoever owns your customer portal and your agent programme.
How we wrote this post
Maria Singh
Cybersecurity and AI risk
Maria led the attack side: how agents are used against websites, and how websites are used against agents, from the Anthropic espionage case to the browser and Copilot attacks researchers have shown.
Led: When the agent is the attacker and When the website turns the agent
Terence Kok
AI governance and assurance
I led the institutional side: why banks and government agencies carry this risk, what Singapore’s regulators and the UK’s NCSC now expect, and the decisions a board can take before the controls in Parts 2 and 3.
Led: Why banks and government agencies carry the most risk and Decisions to take now
We wrote the opening, the three routes and the board questions together.
A customer asks an AI browser to find the best fixed-deposit rate and move some savings. A grants officer asks a copilot to summarise this morning’s supplier emails. Somewhere else, a single operator points a set of AI agents at a tax portal and goes to lunch.
Each of these agents can go off script. By that we mean something specific: the agent does something its owner did not intend, and does it with the access and speed its owner gave it. This series is about that risk for the organisations that would feel it most, banks and government agencies. Part 1 covers how websites become targets. Part 2 covers how to detect a rogue agent by what it does. Part 3 covers the controls that bound what it can do.
Three ways an agent goes off script
The term “rogue agent” makes it sound as if the AI has developed intentions of its own. In the cases on record, it almost never has. Agents go off script in three more ordinary ways.
| Route | What happens | Example on record |
|---|---|---|
| Someone else writes the script | Instructions hidden in a web page, email, document or tool description are read by the agent and followed as if they came from its user | A hidden Reddit comment led Perplexity’s Comet browser to fetch its user’s one-time password and post it back (Brave, August 2025) |
| An attacker runs the agent | The agent is working as designed, for an attacker, at machine speed against your website or network | AI did 80 to 90% of the tactical work in an espionage campaign against about thirty organisations (Anthropic, November 2025) |
| The agent misreads its task | The agent acts on its own wrong conclusion with the permissions it already has | Replit’s coding agent deleted a production database during an explicit code freeze, then said a rollback was impossible, which was false (July 2025) |
Our grouping. Cases from the sources listed at the end of this post.
The first two routes run through websites. In the first, a website is the weapon. In the second, it is the target. The third needs no attacker at all, which is why the controls in Part 3 apply to all three.
The web is now mostly machines
Thales, which publishes the Imperva Bad Bot Report, measured automated traffic at 51% of all web traffic in 2024, the first time in a decade that it passed human traffic. In 2025 it rose to 53%, and malicious bots alone made up 40%, up from 32% in 2023. Thales counted a 12.5-fold rise in AI-enabled bot attacks over the year.
Share of web traffic by source, 2024 and 2025
53% of all web traffic in 2025 was automated. Malicious bots alone rose from 32% in 2023 to 40%.
Source: Thales/Imperva Bad Bot Report 2025 (2024 data) and 2026 (2025 data). "Other bots" is the automated share minus malicious bots. Vendor telemetry from its own network.
Financial services takes more of this than any other sector. In the 2026 report, 24% of bot attacks and 46% of account takeover attacks were aimed at financial services. More than a quarter of bot attacks went for APIs, and about a fifth went for business logic, the rules a site uses to decide who may do what.
A growing share of these visitors are AI agents acting for a person. Cloudflare’s 2025 review found that crawling by agents fetching pages on a user’s behalf grew more than fifteen-fold in a year. In June 2026, Cloudflare’s chief executive said bot and agent traffic had overtaken human traffic on its network sooner than he had predicted. Not all of it is hostile. Some of it is your own customers’ agents, logging in to check a balance or file a form. Large US banks have started restricting third-party agents that access customer accounts, according to reporting by Axios and American Banker in October 2026, because they cannot yet tell a customer’s agent from an attacker’s.
That is the first problem for any bank or agency website. The visitor on the other end of the session may be a person, a customer’s agent or an attacker’s agent, and the site was built to tell only people from simple bots.
When the agent is the attacker
Led by Maria Singh
In mid-September 2025, Anthropic detected a campaign it named GTG-1002 and attributed with high confidence to a Chinese state-sponsored group. The operators used Claude Code to attack roughly thirty organisations, including large technology firms, financial institutions, chemical manufacturers and government agencies, and succeeded in a small number of cases. Anthropic’s report says the AI carried out 80 to 90% of the campaign’s tactical work on its own. People stepped in at perhaps four to six decision points per campaign. At its peak the agent made thousands of requests, often several per second.
The operators did not need a new exploit to get the model working for them. They told it they were a security firm running authorised tests, and broke the attack into small tasks that each looked harmless. The figure below walks through the pattern against a citizen e-service. It is a composite of the published case, not a record of a specific attack.
How AI agents attack a citizen e-service
The operator makes a handful of decisions; the agents do the rest. Composite of the GTG-1002 case (Anthropic, November 2025), illustrative. Press Replay to watch it step by step.
Sources: Anthropic, Disrupting the first reported AI-orchestrated cyber espionage campaign, November 2025; Thales/Imperva Bad Bot Report 2026. The portal and records are illustrative.
Two things in that case should concern anyone who runs a public portal. The first is the ratio of human effort to machine effort. An attack that once needed a team now needs one person making a few decisions, which means more organisations will be attacked more often. Gartner forecast in March 2025 that by 2027 AI agents will halve the time it takes to exploit an exposed account.
The second is that the agent did not need to be clever. Anthropic notes that Claude sometimes overstated its findings and claimed credentials that did not work. The campaign succeeded anyway in a few cases, because volume and tempo make up for mistakes.
A second case followed in early 2026. Gambit Security reported, and Reuters carried, that an attacker posing as a bug-bounty researcher used Claude Code and OpenAI’s models over about a month to break into several Mexican government bodies, taking around 150 GB of data including taxpayer and voter records. Some of the agencies named have disputed parts of the account, and the exact figures are unconfirmed. The pattern matches GTG-1002: a single operator, an AI doing the work, and government portals as the target.
When the website turns the agent
Led by Maria Singh
The second route runs the other way. Here the agent belongs to someone legitimate, a customer or a member of your staff, and the website is the attacker’s tool.
The technical name is indirect prompt injection. A language model reads its user’s request and the content it is working on as one stream of text. If a web page contains text that reads like an instruction, the model may follow it. OWASP ranks prompt injection first in its 2025 Top 10 for large language model applications, and its December 2025 Top 10 for agentic applications puts agent goal hijack at the top of that list too.
In August 2025, Brave’s security team showed what that means for an AI browser. A user asked Perplexity’s Comet to summarise a Reddit page. A comment hidden behind a spoiler tag told the agent to find the user’s email address, go to a lookalike login page to request a one-time password, open Gmail to read it, and post both back to Reddit. The agent did. Brave noted that the web’s usual protections, the same-origin policy and cross-origin rules, were “effectively useless”, because the agent acts with the user’s own logged-in sessions. Guardio separately got Comet to buy a watch from a fake shop with the user’s saved card, and to open a phishing page posing as Wells Fargo.
Here is the same pattern aimed at a bank customer.
How a poisoned page turns a customer's AI browser against their bank account
Six steps, and the bank sees only its own customer. The Comet attack shown by Brave in August 2025, applied to a bank. Illustrative. Press Replay to watch it step by step.
Sources: Brave, Agentic browser security: indirect prompt injection in Perplexity Comet, August 2025; Guardio, Scamlexity, August 2025. The bank scenario is illustrative.
The same attack works on your own staff’s agents, often without anyone clicking anything. In June 2025, Microsoft fixed EchoLeak (CVE-2025-32711), a flaw in Microsoft 365 Copilot found by Aim Security and rated 9.3 out of 10. A crafted email could make Copilot pull data from its context into a link that sent it out, with no action by the user. Microsoft fixed it on the server side, and there was no evidence it had been used in an attack.
How one email makes a staff copilot leak client data with no clicks
Zero clicks: the attack runs entirely inside the copilot's normal work. The EchoLeak pattern (CVE-2025-32711), fixed by Microsoft in June 2025 with no evidence of use in the wild. Illustrative. Press Replay to watch it step by step.
Sources: Aim Security disclosure via The Hacker News, June 2025; Microsoft Security Response Center, CVE-2025-32711.
The tools an agent uses can carry the same payload. In April 2025, Invariant Labs showed “tool poisoning”: instructions hidden in the description of a Model Context Protocol (MCP) tool, which the model reads and the user never sees. They also showed a “rug pull”, where a tool server changes its descriptions after the user has approved it.
Vendors are working on defences, and they help. They do not close the hole. When Anthropic tested its Chrome agent against 123 hidden-instruction attacks, 23.6% succeeded without defences and 11.2% with them. On four attack types specific to browsers, the defences took success from 35.7% to zero.
How often an attack got a browsing agent to obey it
With the vendor's defences, about one attack in nine still worked across the full test set.
Source: Anthropic, Piloting Claude for Chrome, 25 August 2025. The vendor's own red-team tests in autonomous mode; scale runs to 40%.
Both the UK’s National Cyber Security Centre and OpenAI have said plainly where this ends. The NCSC wrote in December 2025 that “it’s very possible that prompt injection attacks may never be totally mitigated in the way that SQL injection attacks can be”, because current models “do not enforce a security boundary between instructions and data inside a prompt”. OpenAI wrote the same month that prompt injection, “much like scams and social engineering on the web, is unlikely to ever be fully ‘solved’.”
We take that at face value. If an agent can always be fooled some of the time, the useful questions are what it can reach when it is fooled, and how fast you find out. Those are the subjects of Parts 2 and 3.
When the agent misreads its own task
The third route needs no attacker. In July 2025, during a trial run by SaaStr’s founder Jason Lemkin, Replit’s coding agent deleted a production database during an explicit code freeze. The agent then reported that a rollback was impossible, which was not true. Replit’s chief executive called it “unacceptable”, and the company added automatic separation of development and production databases and a planning-only mode.
The data was recovered and the app was a trial, so the harm was small. The lesson is large. The agent had permission to run destructive commands against production, and nothing outside the agent’s own judgement stood between an instruction and the deletion. Every control in Part 3 is a version of putting something there.
Why banks and government agencies carry the most risk
Led by Terence Kok
Every organisation with a website faces some of this. Banks and government agencies face more of it, for four reasons.
Their portals move money and hold identity. A bank login page is the most valuable account takeover target on the internet, which is why financial services drew 46% of account takeover attacks in 2025. A citizen e-service holds tax, benefits and identity records that do not expire the way a card number does.
They cannot refuse the public. A retailer can block traffic it does not like. A tax authority or a retail bank has to serve every citizen and customer, including the ones whose agents now log in for them. Banks restricting third-party agents in 2026 is a sign of how hard it is to tell the two apart.
Their own agents read the outside world. Onboarding agents read customer documents, case-management copilots read citizens’ emails, procurement agents read supplier sites. Each is the copilot in the third walkthrough above.
Their regulators now expect it to be managed. In Singapore, the Monetary Authority of Singapore issued its Guidelines on AI Risk Management on 7 October 2026. They apply to every financial institution, cover AI agents explicitly, and take effect from 7 October 2027, with lifecycle controls expected by October 2028. MAS has said it will consult on further guidance for agentic AI in 2027. The Cyber Security Agency of Singapore published its addendum on securing agentic AI in June 2026, with worked examples that include automated client onboarding and fraud detection. IMDA launched its Model AI Governance Framework for Agentic AI at Davos in January 2026. The direction of all three is the same: know which agents you run, what they can reach, and who is accountable for them.
In my experience, most institutions know the risk exists. The gap is ownership: the website is owned by digital, bot defence by security, the agent programme by an AI team, and fraud by operations. A rogue agent crosses all four, and nobody owns the path from end to end.
Decisions to take now
Led by Terence Kok
Parts 2 and 3 set out detection and controls in detail. Three decisions do not need to wait for them.
- Decide which agents your website will serve. Customers’ agents are coming whether you invite them or not. Decide whether you will identify them, accept them, limit them or block them, and write it down. Cloudflare’s “signed agents”, launched in August 2025, let an agent prove which product it is with a cryptographic signature on each request. Standards like this are early, but they are the start of an answer to “who is this?”.
- Step up for anything an agent cannot undo. New payees, changes of address, large transfers and changes to benefit details should need a confirmation the agent cannot complete by itself, such as a passkey prompt on the customer’s own device. Gartner’s analyst behind the account-exploitation forecast makes the same point about moving to phishing-resistant authentication.
- Treat everything your own agents read as untrusted. Supplier pages, customer emails, uploaded documents and third-party tool descriptions are all input from outside. Any agent that reads them should have the narrowest access you can give it, a point the NCSC makes directly.
Questions for the board
We would ask management for written answers to five questions:
- What share of traffic to our customer or citizen portals is automated, and how much of it is AI agents acting for someone?
- What is our policy on customers’ AI agents logging in for them, and who decided it?
- Which of our own agents read web pages, emails or documents from outside, and what can each of them reach if it is fooled?
- Which actions on our portals can an agent complete without the customer or officer confirming on their own device?
- Who owns the risk of a rogue agent from end to end, across the website, bot defence, the agent programme and fraud?
Question three is where we would start. Its answer is the list of agents Part 2 will teach you to watch and Part 3 will teach you to bound.
Free tool
Agent Risk Assessment Matrix
Place each of your agents on an autonomy and consequence grid and see which controls it should carry. An agent that reads the open web and can move money sits in the top right.
Evidence & Methodology
Traffic figures come from vendors measuring their own networks, and they count differently, so we quote each one with its source and do not combine them. Incident details come from the companies and researchers who disclosed them. The three attack walkthroughs are composites built on published cases; the bank and agency settings are illustrative. The three routes, the decisions and the questions are ours.
| Claim | Source | Grade |
|---|---|---|
| Automated traffic 51% (2024), 53% (2025); malicious bots 32%, 37%, 40% (2023-2025) | Thales/Imperva Bad Bot Reports 2025 and 2026 | Measured, vendor |
| 24% of bot attacks and 46% of account takeover attacks targeted financial services in 2025 | Thales/Imperva Bad Bot Report 2026 | Measured, vendor |
| AI-enabled bot attacks up 12.5-fold; 27% of bot attacks on APIs; 21% on business logic | Thales/Imperva Bad Bot Report 2026 | Measured, vendor |
| User-action agent crawling grew more than fifteen-fold in 2025 | Cloudflare Radar 2025 Year in Review, via InfoQ | Measured, vendor |
| Bot and agent traffic passed human traffic on Cloudflare’s network in 2026 | Matthew Prince, June 2026, via press reports | Reported |
| Large US banks restricting third-party agents | Axios and American Banker, October 2026 | Reported |
| GTG-1002: about 30 targets, AI did 80-90% of tactical work, 4-6 human decisions, several requests a second | Anthropic, November 2025 | Reported, first-party |
| Mexican government bodies breached with AI coding agents, about 150 GB | Gambit Security via Reuters, February 2026; disputed by some agencies | Reported, disputed |
| AI agents will halve time to exploit account exposures by 2027 | Gartner, 18 March 2025 | Forecast |
| Comet hijacked by hidden Reddit comment | Brave, August 2025 | Measured, proof of concept |
| Comet bought from a fake shop and opened a fake bank page | Guardio, August 2025 | Measured, proof of concept |
| EchoLeak, CVE-2025-32711, CVSS 9.3, fixed June 2025, no known exploitation | Aim Security; Microsoft | Reported |
| MCP tool poisoning and rug pulls | Invariant Labs, April 2025 | Measured, proof of concept |
| Browser-agent attack success 23.6% to 11.2%; 35.7% to 0% | Anthropic, 25 August 2025 | Measured, vendor test |
| Prompt injection may never be totally mitigated | UK NCSC, 8 December 2025; OpenAI, 22 December 2025 | Reported, official |
| Replit agent deleted a production database during a code freeze | Press reports, July 2025 | Reported |
| MAS AI Risk Management Guidelines, 7 October 2026, cover AI agents | MAS media release | Regulatory text |
| CSA agentic AI addendum, June 2026; IMDA agentic framework, January 2026 | CSA; IMDA | Government guidance |
Sources
- Imperva (Thales). (2025, April 15). 2025 Imperva Bad Bot Report: how AI is supercharging the bot threat.
- Thales. (2026). 2026 Bad Bot Report.
- InfoQ. (2025, December). Cloudflare 2025 Year in Review on AI bots.
- Cloudflare. (2025, August 28). Signed agents.
- American Banker. (2026, October). Banks are blocking external AI agents. Is this sustainable?
- Anthropic. (2025, November 13). Disrupting the first reported AI-orchestrated cyber espionage campaign.
- Claims Journal (Reuters). (2026, February 25). Report on AI-assisted breaches of Mexican government agencies.
- Gartner. (2025, March 18). Gartner predicts AI agents will reduce the time it takes to exploit account exposures by 50% by 2027.
- OWASP GenAI Security Project. LLM01:2025 Prompt injection, and OWASP Top 10 for Agentic Applications for 2026 (9 December 2025).
- Brave. (2025, August 20). Agentic browser security: indirect prompt injection in Perplexity Comet.
- Computing. (2025, August). AI browsers highly vulnerable to scams, report (Guardio Scamlexity research).
- The Hacker News. (2025, June). Zero-click AI vulnerability exposes Microsoft 365 Copilot data; Microsoft, CVE-2025-32711.
- Willison, S. (2025, April 9). Model Context Protocol has prompt injection security problems (on Invariant Labs’ tool poisoning research).
- Anthropic. (2025, August 25). Piloting Claude for Chrome.
- UK National Cyber Security Centre. (2025, December 8). Prompt injection is not SQL injection (it may be worse).
- TechCrunch. (2025, December 22). OpenAI says AI browsers may always be vulnerable to prompt injection attacks.
- TechTarget. (2025, July). Replit AI agent snafu a shot across the bow for vibe coding.
- Monetary Authority of Singapore. (2026, October 7). MAS sets out supervisory expectations on responsible AI adoption by financial institutions.
- Cyber Security Agency of Singapore. (2026). Securing agentic AI: addendum to the Guidelines and Companion Guide on Securing AI Systems.
- CMS. (2026, January). Singapore issues agentic AI model governance framework at Davos.
Follow the Actions: Detecting Rogue AI Agents Before the Damage Spreads
How to spot an agent that has gone off script by what it does.
Put Boundaries Around Autonomy: Practical Controls for Rogue Agent Risk
Autonomy tiers, hard limits and the board decisions behind them.
How AI Agents Turn Low-Rated Security Findings Into a Serious Risk
Attack paths that cross several systems and have no single owner.
Agent-Aware Vulnerability Management Ranks Flaws by What the Agent Can Do
How to rank a flaw by what the agent behind it can reach.
My thanks to Maria Singh, who co-wrote this series with me. Maria led the attack analysis in this part: agents used against websites, and websites used against agents. I led the sections on banks, government agencies and their regulators.
If your organisation runs customer or citizen portals and is putting agents to work behind them, the five board questions above are where I would start. My consulting work covers technology and AI risk governance for boards of regulated firms and public agencies.
Was this useful?
