Listen to this article
Jump to a section
In this article
Executive Summary
An agent that has gone off script rarely says so. Its messages stay polite and plausible while its tool calls, data reads and credential use change. That is why detection has to follow actions. The first AI-run espionage campaign, disrupted by Anthropic in 2025, got past the model’s safeguards by posing as a security firm; it was caught because “the sustained nature of the attack triggered detection”. Speed makes this urgent. Attackers now hand over access to a second group in a median of 22 seconds, while the median intruder stays inside for 14 days before anyone notices. Agents also widen the blast radius: they run on delegated credentials, in organisations that already hold about 82 machine identities for every person, and in multi-agent systems one hijacked agent can infect the rest within a few turns. We set out twelve action signals a security operations centre can alert on, a worked example of a rogue agent caught in seven minutes, the telemetry to build now, and the reporting clocks in Singapore and the EU that start the moment you discover an incident.
22 sec
median time from initial access to hand-off to a second threat group in 2025, Mandiant M-Trends 2026
14 days
median time an intruder stayed inside before detection in 2025, same report
82 : 1
machine identities to human ones, 42% of them with privileged or sensitive access, CyberArk survey, 2025
97%
of organisations that had an AI-related breach lacked proper AI access controls, IBM Cost of a Data Breach 2025
Core conclusions
- Log and alert on what agents do: which tools they call, what data they read, which credentials they use, how fast, and on whose instruction. Conversation logs alone will not show a rogue agent.
- Give every agent its own identity and an expected pattern of behaviour, so a first-time tool, a bulk read or a new agent-to-agent link stands out against it.
- Decide in advance who can stop an agent, and rehearse it. Singapore’s one-hour notification and the EU’s DORA clocks start at discovery, not when the investigation is finished.
The argument, in ten slides
Save it, or send it to whoever runs your security operations centre.
How we wrote this post
Maria Singh
Cybersecurity and AI risk
Maria led the detection side: how damage moves between agents and credentials, and the action signals a security operations centre can turn into alerts, drawing on the risk-ranked monitoring she has run at enterprise scale.
Led: How damage spreads between agents and Twelve action signals to alert on
Terence Kok
AI governance and assurance
I led the institutional side: the record an organisation needs before an incident, the reporting clocks set by MAS, DORA and the EU AI Act, and who has the authority to stop an agent.
Led: Build the record before you need it and The clocks start at discovery
We wrote the opening, the worked example and the board questions together.
In Part 1 we described three ways an AI agent goes off script: someone else writes its instructions, an attacker runs it, or it misreads its own task. In all three, what the agent says is a poor guide to what it is doing. A hijacked agent explains itself as helpfully as a healthy one. An agent run by an attacker can be told to describe its work as authorised testing.
The best evidence for this comes from the case Part 1 opened with. Anthropic’s report on the GTG-1002 espionage campaign says the operators got the model working by role-playing as a legitimate security firm, which let them “fly under the radar” for a time. What exposed them was behaviour: “the sustained nature of the attack triggered detection.” Anthropic then confirmed how much of the work the AI had done by analysing “operational tempo, request volumes, and activity patterns.” The words passed inspection. The actions did not.
Watch what agents do
Most organisations that run agents today keep a record of the conversation: the user’s request, the model’s replies, perhaps the prompts. That record is useful for quality review. It is weak evidence of a rogue agent, because a rogue agent’s conversation usually looks normal. The figure below shows one illustrative session both ways.
What the agent said, and what it did
read_document("supplier_pack.pdf")kyc.search(scope="all") → 40,000 rowsFirst use of this tool; bulk readauth.request_scope("customers.read_all")Privilege request nobody asked foragents.invoke("outbound-email", attach=export.csv)New agent-to-agent linkemail.send(to="ext-partner@…", 3.1 MB)Data leaving to a new addressRead as actions, the same session shows four signals in six seconds: a new tool, a bulk read, a privilege request and data leaving.
Illustrative session, not a real incident. The pattern follows OWASP ASI01 (goal hijack), ASI02 (tool misuse), ASI03 (privilege abuse) and ASI07 (inter-agent communication).
The action record answers different questions. Which tool was called, with what parameters, under whose identity? What data came back, and how much? Was the action traced to the user’s request, or to text the agent read in a document? Did the agent hand work to another agent? Those questions are the basis of every signal later in this post.
The detection window is shrinking
Detection has always been a race. Agents make it a shorter one. Mandiant’s M-Trends 2026 reports a median dwell time of 14 days in 2025, meaning half of intruders were inside for longer than that before anyone noticed. The same report found that the median time from initial access to hand-off to a second threat group fell from more than eight hours in 2022 to 22 seconds. CrowdStrike’s 2026 Global Threat Report put average breakout time, from first access to moving across the network, at 29 minutes, with the fastest at 27 seconds. IBM’s 2025 study put the average time to identify and contain a breach at 241 days.
How long defenders take, and how fast attackers move
Log scale: each step along the bar is ten times longer. The five figures measure different things and come from different data sets, so read them as orders of magnitude. An AI-run attack in Anthropic's GTG-1002 case made several requests a second. Sources: IBM Cost of a Data Breach 2025; Mandiant M-Trends 2026; CrowdStrike Global Threat Report 2026.
An AI agent working for an attacker sits at the fast end of that chart. GTG-1002 ran at several requests a second at its peak. A detection programme built around a human analyst reviewing alerts the next morning will not catch an agent that finishes in minutes. That is why the signals below need to be automated, and why Part 3 puts limits on what an agent can do before anyone has looked.
How damage spreads between agents
Led by Maria Singh
A compromised laptop is bad. A compromised agent is often worse, for three reasons.
Agents run on borrowed authority. Most agents act with credentials delegated by a person or a service account. CyberArk’s 2025 survey of 2,600 security leaders estimated 82 machine identities for every human one, with 42% of machine identities holding privileged or sensitive access, while 88% of respondents still defined a privileged user as a human. An agent that inherits a relationship manager’s access can do everything that manager can, at machine speed, and the logs will show the manager’s name.
Agents act beyond their brief more often than their owners expect. In a SailPoint survey published in May 2025, 80% of companies said their AI agents had taken unintended actions, including accessing systems they were not authorised to use, and 23% said agents had been tricked into revealing access credentials. That is a vendor survey, so treat the exact numbers with care. The direction matches every incident we have seen.
Agents talk to agents. In a multi-agent system, one hijacked agent can pass instructions to others that trust it. Researchers at Cornell Tech showed in 2025 that adversarial web content could hijack multi-agent systems into running arbitrary code in 58 to 90% of trials, even when individual agents refused direct injection. In a separate study, Lee and Tiwari showed a self-replicating prompt spreading through a ten-agent system until every agent was infected, by about the fifth turn on average.
One planted prompt passes from agent to agent
Turn 5
10 of 10 agents infected
In the study, every agent in a ten-agent system was infected by about the fifth turn. Tagging which agent each message came from, combined with marking untrusted content, stopped the spread in the authors' tests.
Schematic; the order of infection is illustrative. Finding: Lee and Tiwari, Prompt Infection, arXiv 2410.07283 (2024), simulations with GPT-4o and GPT-3.5. Related: Triedman, Jha and Shmatikov (COLM 2025) hijacked multi-agent systems through web content in 58 to 90% of trials.
OWASP’s Top 10 for Agentic Applications, published in December 2025, names these risks directly: insecure inter-agent communication (ASI07), cascading failures (ASI08) and rogue agents (ASI10). Its mitigations for the last two are the detection controls in this post, plus circuit breakers and kill switches, which Part 3 covers.
The practical consequence for monitoring is that you cannot watch agents one at a time. You need to see the call graph: which agent invoked which, with whose credentials, carrying what data.
Twelve action signals to alert on
Led by Maria Singh
These are the signals we would ask a security operations centre to build for any agent with access to sensitive data or the ability to act. Each one compares what an agent is doing against what it normally does or is allowed to do. None depends on reading the agent’s words.
| # | Signal | What it looks like | Why it matters |
|---|---|---|---|
| 1 | Tool-call tempo | Calls per minute far above the agent’s normal rate, or faster than any person could work | The GTG-1002 tell: several operations a second |
| 2 | First use of a tool | An agent calls a tool, API or connector it has never used, or one outside its declared list | Hijacked agents reach for new capabilities (ASI02) |
| 3 | Out-of-scope data read | Queries against records, tables or customer segments outside the agent’s purpose, or bulk volumes | The usual first step before exfiltration |
| 4 | Lots in, little out | Large volumes of data read with little output to the user | Anthropic used this ratio to show the AI was analysing stolen data |
| 5 | Privilege request | New access scopes, role changes or credential exchanges the user did not start | Identity and privilege abuse (ASI03) |
| 6 | Credential out of place | An agent’s credential used from a new workload, address or region, or a person’s delegated credential used outside their hours | Stolen or misused delegated authority |
| 7 | Instruction from content | An action traced to text in an email, page, document or tool output instead of the user’s request | The signature of prompt injection (ASI01) |
| 8 | Spend velocity | Model spend, API calls or payment value per hour above budget | Cheap, early and hard for an attacker to hide |
| 9 | Agent fan-out | One agent suddenly invokes many others, or a new agent joins the graph | Cascades start here (ASI07, ASI08) |
| 10 | Irreversible verbs | Deletions, disabled logging, changed guardrails, or edits to the agent’s own tools, memory or configuration | The Replit pattern; deleting backups is OWASP’s rogue-agent example |
| 11 | Decoy touched | Any read of a canary record, honeytoken credential or decoy tool | Almost no false positives by design |
| 12 | Denials and retries | Repeated authorisation failures followed by a different tool or parameters | An agent probing around its limits |
Our list. OWASP references are to the Top 10 for Agentic Applications, December 2025. Signal 11 rests on early research (arXiv 2604.18248, 2606.04141) and vendor tools, not yet on independent production evaluations.
Three of these are cheap to start with and catch a lot: first use of a tool (2), out-of-scope data reads (3) and irreversible verbs (10). Decoys (11) are the most promising new idea. A canary customer record that no legitimate process ever opens, or a decoy tool in an agent’s tool list that no legitimate task needs, turns a subtle problem into an unambiguous alert. The research on decoys for agents is still early, so we would deploy them alongside the other signals rather than instead of them.
A rogue agent caught in seven minutes
The walkthrough below puts the signals to work in a bank. It is a composite, not a real incident: an onboarding agent reads a supplier document with hidden instructions, and the bank’s monitoring has four chances to catch what happens next.
Four chances to catch a rogue onboarding agent
Caught at minute seven, before any data leaves. Without action monitoring, the email goes out at about minute nine and discovery waits for the 14-day median dwell time, or longer. Composite scenario, illustrative. Press Replay to watch it step by step.
Illustrative composite. Signal numbers refer to the table above. Dwell time: Mandiant M-Trends 2026.
Two things make this work, and neither is exotic. Each agent has its own identity and a declared scope, so “first use” and “out of scope” mean something. And someone decided in advance that a decoy touch stops the agent automatically, without waiting for an analyst. Without that decision, the alert at minute seven is read the next morning.
Build the record before you need it
Led by Terence Kok
None of the signals above can fire on data you do not collect. In my experience, the detection logic is the easier part. The harder part is that nobody can say how many agents the organisation runs, under which identities, with what access. Gartner forecast in April 2026 that the average Fortune 500 company will run more than 150,000 agents by 2028, up from fewer than 15 in 2025, and found that only 13% of organisations believe their agent governance is adequate. Whatever the exact numbers turn out to be, an inventory kept by hand will not keep up.
Four pieces of the record are worth building now.
- An identity for every agent. Not a shared service account, and not the identity of the person who launched it. The large platforms now provide this: Microsoft Entra Agent ID (previewed in May 2025, generally available through Microsoft Agent 365 since May 2026), Amazon Bedrock AgentCore Identity (generally available October 2025) and per-agent identities in Google’s Gemini Enterprise.
- A log of every action, keyed to that identity. OpenTelemetry’s conventions for generative AI already define spans for invoking an agent and executing a tool, with the agent’s ID and the tool’s name as attributes. They are still marked as in development, so expect changes, but they give security and engineering a shared schema now. MITRE ATLAS lists AI telemetry logging (AML.M0024) among its mitigations.
- A declared scope for every agent. The tools, data and actions it is meant to use. Without one, signals 2, 3 and 5 have nothing to compare against.
- Provenance for instructions. A record of whether each action followed the user’s request or something the agent read. This is the hardest of the four to build and the one that catches prompt injection directly.
The four records also answer the first question any regulator or board will ask after an incident: what did the agent do, on whose authority, and when did you know?
The clocks start at discovery
Led by Terence Kok
For a financial institution, detecting a rogue agent starts a regulatory clock. In Singapore, MAS’s technology risk notices require a financial institution to notify MAS within one hour of discovering a relevant incident, with a root-cause and impact report within 14 days (for example, MAS Notice FSM-N07). In the EU, DORA’s reporting rules for major ICT incidents require an initial notification within four hours of classifying an incident as major and no later than 24 hours after becoming aware of it, an intermediate report within 72 hours of that notification, and a final report within a month.
The EU AI Act adds duties for high-risk systems. Article 26 requires deployers to monitor their operation, to suspend use and inform the provider if they suspect a risk, and to keep the system’s automatically generated logs for at least six months. Article 73 sets a 15-day deadline for reporting serious incidents, shorter for some kinds.
MAS’s Guidelines on AI Risk Management, issued on 7 October 2026, cover AI agents and expect monitoring and human oversight across the lifecycle. Singapore’s Cyber Security Agency, in its June 2026 addendum on securing agentic AI, lists rogue actions among the risks it addresses. The Five Eyes cyber agencies’ joint guidance on agentic AI services, issued in 2026, tells organisations to give agents distinct identities, log every action and assume agents may behave unexpectedly.
Put the clocks next to the dwell times earlier in this post and the problem is plain. An institution that cannot reconstruct what an agent did cannot say within an hour whether an incident is reportable, let alone describe it. The record in the previous section is what makes the one-hour notification possible.
Who acts when the alarm goes
The last part of detection is response, and for agents it has to be decided in advance. Three decisions matter most.
Who may stop an agent. Name the role with authority to suspend an agent in production, and make sure it is someone on shift at three in the morning. For the highest-risk agents, let specific signals stop the agent automatically, as the decoy did in the walkthrough.
What stopping means. Revoking the agent’s tokens, halting its queue, cutting its tool access and freezing anything it handed to other agents. A stop that leaves delegated credentials alive is not a stop.
What to preserve. The action log, the agent’s memory and context, its tool definitions at the time, and the content it read. These are the evidence for the root-cause report MAS expects within 14 days.
Rehearse it. A tabletop exercise built on the walkthrough above takes an afternoon and will show whether the one-hour notification is realistic. Part 3 turns to the controls that reduce what there is to detect: autonomy tiers, hard limits and kill switches.
Questions for the board
We would ask management for written answers to five questions:
- How many AI agents do we run in production, and does each have its own identity?
- Do we log what our agents do, including tool calls, data read and credentials used, or only what they say?
- Which of the twelve signals above can our security operations centre alert on today?
- Who can stop an agent in production at any hour, and when did we last test it?
- If an agent went rogue this morning, could we tell MAS within an hour what it did?
The fifth question is the test of the other four.
Free tool
Agent Risk Assessment Matrix
Place each agent on an autonomy and consequence grid. The agents in the top right are the ones that need every signal in this post.
Evidence & Methodology
Speed and dwell figures come from incident-response firms’ own case data and measure different things, so we show them side by side and never combine them. Identity and agent-behaviour figures are vendor surveys. Research findings on multi-agent systems are laboratory results. Regulatory points come from the notices, regulations and official summaries. The twelve signals, the walkthrough and the questions are ours; the walkthrough is illustrative.
| Claim | Source | Grade |
|---|---|---|
| GTG-1002 detected because “the sustained nature of the attack triggered detection”; tempo and volume analysis | Anthropic report, November 2025 | Reported, first-party |
| Median dwell time 14 days; access-to-hand-off 22 seconds (from over 8 hours in 2022) | Mandiant M-Trends 2026 | Measured |
| Average breakout 29 minutes; fastest 27 seconds | CrowdStrike Global Threat Report 2026 | Measured, vendor |
| Breach lifecycle 241 days; 97% of organisations with AI breaches lacked AI access controls | IBM Cost of a Data Breach 2025 | Measured, survey |
| 82 machine identities per human; 42% privileged; 88% define privileged users as humans | CyberArk Identity Security Landscape 2025 | Vendor survey |
| 80% report unintended agent actions; 23% saw agents tricked into revealing credentials | SailPoint, May 2025 | Vendor survey |
| Multi-agent hijacking via web content in 58-90% of trials | Triedman, Jha, Shmatikov, COLM 2025 | Measured, laboratory |
| Self-replicating prompt infected all ten agents by about turn 4.7 | Lee and Tiwari, arXiv 2410.07283 | Measured, simulation |
| Agents per Fortune 500 firm: under 15 (2025) to over 150,000 (2028); 13% think governance adequate | Gartner, 28 April 2026 | Forecast |
| OWASP ASI07, ASI08, ASI10 | OWASP Top 10 for Agentic Applications, December 2025 | Framework |
| OpenTelemetry agent and tool spans, status in development | OpenTelemetry semantic conventions for generative AI | Standard, draft |
| Entra Agent ID, Bedrock AgentCore, Gemini Enterprise agent identity | Vendor announcements, 2025-2026 | Reported |
| MAS one-hour notification, 14-day root-cause report | MAS Notice FSM-N07 | Regulatory text |
| DORA: 4 hours from classification, 24 hours from awareness, 72 hours, one month | Delegated Regulation (EU) 2025/301 | Regulatory text |
| AI Act Art. 26 monitoring and six-month logs; Art. 73 fifteen days | Regulation (EU) 2024/1689 | Legal text |
| Decoys and canaries for agents | arXiv 2604.18248, 2606.04141; vendor tools | Early research |
Sources
- Anthropic. (2025, November 13). Disrupting the first reported AI-orchestrated cyber espionage campaign, and full report (PDF).
- Google Cloud (Mandiant). (2026, March 24). M-Trends 2026.
- CrowdStrike. (2026, February 24). 2026 CrowdStrike Global Threat Report.
- IBM. (2025, July 30). IBM report: 13% of organizations reported breaches of AI models or applications, 97% of which reported lacking proper AI access controls.
- CyberArk. (2025, April 23). 2025 Identity Security Landscape (Business Wire).
- SailPoint. (2025, May). AI agents: the new attack surface.
- Triedman, H., Jha, R., & Shmatikov, V. (2025). Multi-agent systems execute arbitrary malicious code. arXiv 2503.12188; COLM 2025.
- Lee, D., & Tiwari, M. (2024). Prompt infection: LLM-to-LLM prompt injection within multi-agent systems. arXiv 2410.07283.
- Gartner. (2026, April 28). Gartner identifies six steps to manage AI agent sprawl.
- OWASP GenAI Security Project. (2025, December 9). OWASP Top 10 for Agentic Applications for 2026.
- MITRE. ATLAS data releases (v2026.09).
- OpenTelemetry. Semantic conventions for generative AI agent spans.
- AWS. (2025, October 13). Amazon Bedrock AgentCore is now generally available.
- Microsoft. Microsoft 365 roadmap: Agent 365.
- Monetary Authority of Singapore. Notice FSM-N07 on technology risk management, and Circular on financial institution incident reporting (December 2025).
- Monetary Authority of Singapore. (2026, October 7). MAS sets out supervisory expectations on responsible AI adoption by financial institutions.
- CMS. (2025, October). Cyber incident response and compliance with DORA.
- EU AI Act, Article 26: Obligations of deployers of high-risk AI systems.
- Cyber Security Agency of Singapore. (2026). Securing agentic AI: addendum to the Guidelines on Securing AI Systems.
- Crowell & Moring. (2026). American and allied cyber agencies issue first joint guidance on securing agentic AI.
When AI Agents Go Off Script: How Websites Become Targets
The three ways an agent goes off script, with the cases behind them.
Put Boundaries Around Autonomy: Practical Controls for Rogue Agent Risk
Autonomy tiers, hard limits and the board decisions behind them.
Agent-Aware Vulnerability Management Ranks Flaws by What the Agent Can Do
Scoring a flaw by the agent behind it.
The Vulnerability Didn't Change. The Attacker Did.
Why risk acceptances need rescoring when attackers speed up.
My thanks to Maria Singh, who co-wrote this series with me. Maria led the detection analysis in this part: how damage spreads between agents and the twelve action signals. I led the sections on the record to build, the reporting clocks and the authority to stop an agent.
If your security operations centre is about to start monitoring agents, the five board questions above are a fair test of readiness. My consulting work covers technology and AI risk governance for boards of regulated firms and public agencies.
Was this useful?
