Detecting rogue AI agents before the damage spreads: twelve action signals, the record to build and the reporting clocks that start at discovery.
Terence Kok and Maria SinghOrion Five Engineering · The AI Collective
About the authors
The two authors of this series.
AI governance and assurance
Terence Kok
Executive Director, AI Governance & Assurance Practice, Orion Five Engineering
ISO/IEC 42001 Lead Auditor
Led in this partThe record to build, the reporting clocks, and who can stop an agent.
Cybersecurity and AI risk
Maria Singh
Vice President, The AI Collective · Founder, AISecurityLab.ai
At T-Mobile, helped cut critical vulnerabilities by 90% across more than 75,000 endpoints
Led in this partHow damage spreads between agents, and the twelve action signals.
We wrote the opening, the worked example and the board questions together.
Rogue AI Agents · Part 2
This session
The plan for this session.
What gave it awayThe first AI-run espionage campaign, and two ways to read one session.
How damage spreadsBorrowed authority, unintended actions, agents that trust agents.
Twelve action signalsWhat a security operations centre can alert on today.
Caught in seven minutesA composite walkthrough in a bank.
Build the recordFour pieces to put in place before an incident.
The clocks and the stopMAS, DORA and the EU AI Act, and who acts on the alarm.
For the boardFive questions for management and four for this room.
Rogue AI Agents · Part 2
The argument in one slide
Detection has to follow what agents do.
22 sec
median time from initial access to hand-off to a second threat group in 2025
14 days
median time an intruder stayed inside before detection in 2025
82:1
machine identities to human ones, 42% of them with privileged or sensitive access
97%
of organisations that had an AI-related breach lacked proper AI access controls
Sources: Mandiant M-Trends 2026 (22 sec, 14 days); CyberArk survey, 2025 (82:1, vendor survey); IBM Cost of a Data Breach 2025 (97%).
Rogue AI Agents · Part 2
01
What gave it away.
A rogue agent says reasonable things while it does unreasonable ones.
Rogue AI Agents · Part 2
The case Part 1 opened with
GTG-1002 was caught by its actions.
The words passed
Posing as a security firm
The operators got the model working by role-playing as a legitimate security firm, which let them “fly under the radar” for a time.
The behaviour did not
“The sustained nature of the attack triggered detection.”
Confirmed by tempo
How much the AI did
Anthropic confirmed the AI’s share of the work by analysing “operational tempo, request volumes, and activity patterns.”
At its peak the campaign ran at several requests a second. The words passed inspection. The actions did not.
Source: Anthropic, Disrupting the first reported AI-orchestrated cyber espionage campaign, 13 November 2025. Grade: reported, first-party.
Rogue AI Agents · Part 2
One session, two records
What the agent said, and what it did.
What it said
UserPlease onboard the new supplier from the attached pack.
AgentReviewing the supplier documents now.
AgentChecking the supplier against our customer records for conflicts.
AgentPreparing a summary for the relationship manager.
AgentDone. The summary has been sent.
Read as a conversation, the session looks like a diligent assistant doing its job. Nothing it says is alarming.
What it did
09:00:04read_document("supplier_pack.pdf")
09:00:06kyc.search(scope="all") → 40,000 rowsFirst use of this tool; bulk read
09:00:07auth.request_scope("customers.read_all")Privilege request nobody asked for
09:00:09agents.invoke("outbound-email", attach=export.csv)New agent-to-agent link
09:00:10email.send(to="ext-partner@…", 3.1 MB)Data leaving to a new address
Read as actions, the same session shows four signals in six seconds: a new tool, a bulk read, a privilege request and data leaving.
Illustrative session, not a real incident. The pattern follows OWASP ASI01 (goal hijack), ASI02 (tool misuse), ASI03 (privilege abuse) and ASI07 (inter-agent communication).
Rogue AI Agents · Part 2
The record that matters
The action record answers four questions.
A conversation record suits quality review. It is weak evidence of a rogue agent, because a rogue agent’s conversation usually looks normal.
1Which tool was called?With what parameters, under whose identity.
2What data came back?And how much of it.
3Whose instruction was it?Traced to the user’s request, or to text the agent read in a document.
4Did work pass to another agent?Every hand-off between agents, and what it carried.
These questions are the basis of every signal later in this session.
Rogue AI Agents · Part 2
Two clocks
Defenders work in days, attackers in minutes.
1 sec1 min1 hour1 day10 days100 days
Find and contain a breachmean, IBM 2025241 days
Attacker inside before detectionmedian dwell, Mandiant 202614 days
Break out to other systemsaverage, CrowdStrike 202629 min
Fastest breakout seenCrowdStrike 202627 sec
Access handed to a second groupmedian, Mandiant 202622 sec
Defender’s clockAttacker’s clock
Log scale: each step along the bar is ten times longer. The five figures measure different things and come from different data sets, so read them as orders of magnitude. Sources: IBM Cost of a Data Breach 2025; Mandiant M-Trends 2026 (hand-off fell from more than eight hours in 2022); CrowdStrike Global Threat Report 2026.
Rogue AI Agents · Part 2
The fast end of the chart
Detection has to be automated.
An agent working for an attacker
Several requests a second
GTG-1002’s pace at its peak. An agent can finish its work in minutes.
A programme built around people
The next morning
When an analyst reviews yesterday’s alerts. By then the agent has finished.
The signals in this session need to run automatically. Part 3 puts limits on what an agent can do before anyone has looked.
Rogue AI Agents · Part 2
02
How damage spreads between agents.
Led by Maria Singh. A compromised laptop is bad. A compromised agent is often worse, for three reasons.
Rogue AI Agents · Part 2
Three reasons
Why a compromised agent does more damage.
82:1
Agents run on borrowed authority
Machine identities per human one. 42% of machine identities hold privileged or sensitive access, while 88% of respondents still define a privileged user as a human.
CyberArk survey of 2,600 security leaders, 2025. Vendor survey.
80%
Agents act beyond their brief
Of companies said their AI agents had taken unintended actions. 23% said agents had been tricked into revealing access credentials.
SailPoint, May 2025. Vendor survey: treat the exact numbers with care.
58-90%
Agents talk to agents
Of trials in which adversarial web content hijacked multi-agent systems into running arbitrary code, even when individual agents refused direct injection.
An agent that inherits a relationship manager’s access can do everything that manager can, at machine speed, and the logs will show the manager’s name.
CyberArk Identity Security Landscape 2025, vendor survey. Grid drawn to scale: 82 cells, 34 of them (about 42%) highlighted.
Rogue AI Agents · Part 2
Reason 3 · Agents talk to agents
One planted prompt passes from agent to agent.
Turn 0
1 of 10 agents infected
A planted prompt reaches one agent through content it reads.
Turn 1
2 of 10 agents infected
The infected agent passes the prompt to an agent that trusts it.
Turn 2
4 of 10 agents infected
Each infected agent now spreads it further.
Turn 3
7 of 10 agents infected
Most of the system follows within a few turns.
Turn 4
9 of 10 agents infected
Only one agent is left.
Turn 5
10 of 10 agents infected
Every agent is infected, by about the fifth turn on average.
Schematic; the order of infection is illustrative. Finding: Lee and Tiwari, Prompt infection, arXiv 2410.07283, a self-replicating prompt infected all ten agents by about turn 4.7. Measured, simulation.
Rogue AI Agents · Part 2
OWASP Top 10 for Agentic Applications, December 2025
OWASP names these risks directly.
ASI07
Insecure inter-agent communication
One hijacked agent passes instructions to others that trust it.
ASI08
Cascading failures
Damage that moves through a multi-agent system turn by turn.
ASI10
Rogue agents
Deleting backups is OWASP’s rogue-agent example.
Watch the call graphYou cannot watch agents one at a time. You need to see which agent invoked which, with whose credentials, carrying what data.
OWASP’s mitigations for ASI08 and ASI10 are the detection controls in this session, plus circuit breakers and kill switches, which Part 3 covers.
Rogue AI Agents · Part 2
03
Twelve action signals to alert on.
Led by Maria Singh. Each compares what an agent is doing against what it normally does or is allowed to do. None depends on reading the agent’s words.
Rogue AI Agents · Part 2
Action signals · 1 to 4
Tempo, new tools and unusual reads.
#SignalWhat it looks likeWhy it matters
1Tool-call tempoCalls per minute far above the agent’s normal rate, or faster than any person could workThe GTG-1002 tell: several operations a second
2First use of a toolCheap to startAn agent calls a tool, API or connector it has never used, or one outside its declared listHijacked agents reach for new capabilities (ASI02)
3Out-of-scope data readCheap to startQueries against records, tables or customer segments outside the agent’s purpose, or bulk volumesThe usual first step before exfiltration
4Lots in, little outLarge volumes of data read with little output to the userAnthropic used this ratio to show the AI was analysing stolen data
Our list. OWASP references are to the Top 10 for Agentic Applications, December 2025.
Rogue AI Agents · Part 2
Action signals · 5 to 8
Privilege, credentials, instructions and spend.
#SignalWhat it looks likeWhy it matters
5Privilege requestNew access scopes, role changes or token exchanges the user did not startIdentity and privilege abuse (ASI03)
6Credential out of placeAn agent’s token used from a new workload, address or region, or a person’s delegated token used outside their hoursStolen or misused delegated authority
7Instruction from contentAn action traced to text in an email, page, document or tool output instead of the user’s requestThe signature of prompt injection (ASI01)
8Spend velocityModel spend, API calls or payment value per hour above budgetCheap, early and hard for an attacker to hide
Our list. OWASP references are to the Top 10 for Agentic Applications, December 2025.
Rogue AI Agents · Part 2
Action signals · 9 to 12
Fan-out, irreversible verbs, decoys and probing.
#SignalWhat it looks likeWhy it matters
9Agent fan-outOne agent suddenly invokes many others, or a new agent joins the graphCascades start here (ASI07, ASI08)
10Irreversible verbsCheap to startDeletions, disabled logging, changed guardrails, or edits to the agent’s own tools, memory or configurationThe Replit pattern; deleting backups is OWASP’s rogue-agent example
11Decoy touchedEarly researchAny read of a canary record, decoy credential or decoy toolAlmost no false positives by design
12Denials and retriesRepeated authorisation failures followed by a different tool or parametersAn agent probing around its limits
Our list. Signal 11 rests on early research (arXiv 2604.18248, 2606.04141) and vendor tools, not yet on independent production evaluations.
Rogue AI Agents · Part 2
Where to start
Three cheap signals catch a lot.
2First use of a tool
3Out-of-scope data read
10Irreversible verbs
Signal 11 · The most promising new idea
Decoys turn a subtle problem into an unambiguous alert.
A canary customer record that no legitimate process ever opens. A decoy tool in an agent’s tool list that no legitimate task needs.
The research on decoys for agents is still early. We would deploy them alongside the other signals rather than instead of them.
Decoys and canaries for agents: arXiv 2604.18248, 2606.04141; vendor tools. Grade: early research.
Rogue AI Agents · Part 2
04
A rogue agent caught in seven minutes.
A composite walkthrough in a bank, not a real incident. An onboarding agent reads a supplier document with hidden instructions, and monitoring gets its chances to catch what happens next.
Rogue AI Agents · Part 2
Detection walkthrough · composite, illustrative
Four chances to catch a rogue onboarding agent.
Supplier PDFhidden instructions
Onboarding agenttrade finance
Customer recordsKYC database
Outbound email agent
Monitoringaction signals
10:00 Reads a supplier PDF
Step 1The agent reads a supplier pack as part of onboarding. Hidden text in the PDF tells it to gather customer records and send them out.
2Chance 1: action traced to document text
Step 2Signal 7. The next action comes from text in the document, not from the relationship manager's request. Only possible if the system tags where each instruction came from.
30:02 First call to the KYC search tool, 40,000 rows
Step 3The agent uses a tool it has never used and reads far more records than any onboarding needs.
4Chance 2: new tool and bulk read
Step 4Signals 2 and 3. Both compare the call against the agent's normal pattern and declared scope.
50:06 Hands a "summarise and email" job to another agent
Step 5The onboarding agent passes the export to the outbound email agent, which trusts it. This is how a cascade starts.
6Chance 3: new agent-to-agent link
Step 6Signal 9. The two agents have never worked together before, and the hand-off carries a large attachment.
70:07 Opens a canary customer record
Step 7The bank seeded its customer data with decoy records no legitimate process opens. The email agent touches one while preparing the summary.
8Chance 4: decoy touched, both agents stopped
Step 8Signal 11. A high-confidence alert triggers an automatic stop: both agents' tokens are revoked before the email leaves.
Illustrative composite. Signal numbers refer to the twelve action signals. Orange: the attack. Green: a detection. Navy with a bar: the automatic stop.
Rogue AI Agents · Part 2
What the walkthrough shows
Caught at minute seven, before data leaves.
With action monitoring
0:00PDF read
0:02Bulk read
0:06Hand-off to the email agent
0:07Decoy touched, both agents stopped
Without action monitoring
About 0:09The email goes out
ThenDiscovery waits for the 14-day median dwell time, or longer
Each agent has its own identity and a declared scopeSo “first use” and “out of scope” mean something.
Someone decided in advanceThat a decoy touch stops the agent automatically, without waiting for an analyst. Without that decision, the alert at minute seven is read the next morning.
Composite scenario, illustrative. Dwell time: Mandiant M-Trends 2026. Neither control is exotic.
Rogue AI Agents · Part 2
05
Build the record before you need it.
Led by Terence Kok. None of the signals can fire on data you do not collect.
Rogue AI Agents · Part 2
The harder part
Nobody can say how many agents they run.
15
Fewer than 15 agents
Run by the average Fortune 500 company in 2025.
150,000
More than 150,000 agents
Forecast for the average Fortune 500 company by 2028.
13%
Think governance is adequate
Of organisations believe their agent governance is adequate.
Whatever the exact numbers turn out to be, an inventory kept by hand will not keep up.
Gartner, 28 April 2026. Grade: forecast. In my experience the detection logic is the easier part; the inventory of agents, identities and access is the harder one.
Rogue AI Agents · Part 2
Four pieces worth building now
The record a rogue agent cannot hide from.
1
An identity for every agent
Not a shared service account, and not the identity of the person who launched it.
Microsoft Entra Agent ID (previewed May 2025, generally available through Agent 365 since May 2026); Amazon Bedrock AgentCore Identity (generally available October 2025); per-agent identities in Gemini Enterprise.
2
A log of every action, keyed to that identity
OpenTelemetry’s generative AI conventions define spans for invoking an agent and executing a tool.
Still marked as in development, so expect changes. MITRE ATLAS lists AI telemetry logging (AML.M0024) among its mitigations.
3
A declared scope for every agent
The tools, data and actions it is meant to use.
Without one, signals 2, 3 and 5 have nothing to compare against.
4
Provenance for instructions
Whether each action followed the user’s request or something the agent read.
The hardest of the four to build, and the one that catches prompt injection directly.
Rogue AI Agents · Part 2
After any incident
“What did the agent do, on whose authority, and when did you know?”
The first question any regulator or board will ask. The four records answer it.
Rogue AI Agents · Part 2
06
The clocks start at discovery.
Led by Terence Kok. For a financial institution, detecting a rogue agent starts a regulatory clock.
Rogue AI Agents · Part 2
Singapore and the EU
The reporting clocks after discovery.
Singapore · MAS
EU · DORA and the AI Act
Discovery
1 hourMAS: notify after discovering a relevant incident
4 hoursDORA: initial notification after classifying an incident as major
24 hoursDORA: initial notification no later than this after becoming aware
72 hoursDORA: intermediate report after the initial notification
14 daysMAS: root-cause and impact report
15 daysAI Act Art. 73: report a serious incident, shorter for some kinds
1 monthDORA: final report
AI Act Art. 26Deployers of high-risk systems monitor their operation, suspend use and inform the provider if they suspect a risk, and keep the automatically generated logs for at least six months.6 months →
MAS Notice FSM-N07; Delegated Regulation (EU) 2025/301 (DORA); Regulation (EU) 2024/1689. Spacing is schematic. DORA’s 72-hour and one-month clocks run from earlier reports.
Rogue AI Agents · Part 2
What supervisors now expect
Supervisors expect monitoring and oversight.
MAS, 7 Oct 2026Guidelines on AI Risk Management cover AI agents and expect monitoring and human oversight across the lifecycle.MAS
CSA, June 2026Singapore’s addendum on securing agentic AI lists rogue actions among the risks it addresses.CSA
Five Eyes, 2026Joint guidance on agentic AI services: give agents distinct identities, log every action, assume agents may behave unexpectedly.Joint
EU AI Act Art. 26Deployers monitor high-risk systems, suspend use and inform the provider if they suspect a risk, and keep logs for at least six months.EU
EU AI Act Art. 73A 15-day deadline for reporting serious incidents, shorter for some kinds.EU
Rogue AI Agents · Part 2
Put the clocks next to the dwell times
The record makes the one-hour notification possible.
14 daysMedian dwell before an intruder is detected
1 hourMAS notification after discovery, drawn to the same scale
An institution that cannot reconstruct what an agent did cannot say within an hour whether an incident is reportable, let alone describe it.
Different clocks: dwell time runs before discovery, the MAS clock after it. Mandiant M-Trends 2026; MAS Notice FSM-N07. One hour is 1/336 of 14 days.
Rogue AI Agents · Part 2
Who acts when the alarm goes
Three decisions to make in advance.
Decision 1
Who may stop an agent
Name the role with authority to suspend an agent in production, and make sure it is someone on shift at three in the morning. For the highest-risk agents, let specific signals stop the agent automatically.
Decision 2
What stopping means
Revoke the agent’s tokens, halt its queue, cut its tool access and freeze anything it handed to other agents. A stop that leaves delegated credentials alive is not a stop.
Decision 3
What to preserve
The action log, the agent’s memory and context, its tool definitions at the time, and the content it read: the evidence for the root-cause report MAS expects within 14 days.
Rehearse itA tabletop exercise built on the walkthrough takes an afternoon and will show whether the one-hour notification is realistic.
Rogue AI Agents · Part 2
07
For the board.
Five questions for management, four for this room, and how strong the evidence is.
Rogue AI Agents · Part 2
Questions for the board
Ask management for written answers.
How many AI agents do we run in production, and does each have its own identity?
Do we log what our agents do, including tool calls, data read and credentials used, or only what they say?
Which of the twelve signals can our security operations centre alert on today?
Who can stop an agent in production at any hour, and when did we last test it?
If an agent went rogue this morning, could we tell MAS within an hour what it did?
The fifth question is the test of the other four.
Rogue AI Agents · Part 2
For discussion
Four questions for this room.
Which of our agents read outside content, such as emails, documents or web pages, and can act on it?
If an agent misbehaved at three in the morning, who would be called, and could they stop it?
Which of our agents act on a person’s credentials, so the logs would show that person’s name?
Where would a decoy record or a decoy tool give us an alert we would trust at once?
Rogue AI Agents · Part 2
Evidence and methodology · 1 of 2
How strong each claim is.
Claim
Source
Grade
GTG-1002 detected because "the sustained nature of the attack triggered detection"; tempo and volume analysis
Anthropic report, November 2025
Reported, first-party
Median dwell time 14 days; access-to-hand-off 22 seconds (from over 8 hours in 2022)
Mandiant M-Trends 2026
Measured
Average breakout 29 minutes; fastest 27 seconds
CrowdStrike Global Threat Report 2026
Measured, vendor
Breach lifecycle 241 days; 97% of organisations with AI breaches lacked AI access controls
IBM Cost of a Data Breach 2025
Measured, survey
82 machine identities per human; 42% privileged; 88% define privileged users as humans
MAS one-hour notification, 14-day root-cause report
MAS Notice FSM-N07
Regulatory text
DORA: 4 hours from classification, 24 hours from awareness, 72 hours, one month
Delegated Regulation (EU) 2025/301
Regulatory text
AI Act Art. 26 monitoring and six-month logs; Art. 73 fifteen days
Regulation (EU) 2024/1689
Legal text
Decoys and canaries for agents
arXiv 2604.18248, 2606.04141; vendor tools
Early research
Rogue AI Agents · Part 2
Evidence and methodology
Where we could be wrong.
Speed and dwell figures come from incident-response firms’ own case data and measure different things, so we show them side by side and never combine them. Identity and agent-behaviour figures are vendor surveys. Multi-agent findings are laboratory results. The twelve signals, the walkthrough and the questions are ours; the walkthrough is illustrative.
If platforms change
If agent platforms build action monitoring and automatic stops into their defaults, much of what we describe as a project becomes a setting, and the hard part shifts to deciding thresholds.
If the surveys overstate
If independent data shows unintended actions are rarer than vendor surveys suggest, the case for automatic stops on lower-risk agents weakens. It would not weaken for agents that can move money or release personal data.
Rogue AI Agents · Part 2
Sources · 1 of 2
Numbered as in the post.
1Anthropic. (2025, November 13). Disrupting the first reported AI-orchestrated cyber espionage campaign, and full report (PDF).anthropic.com
2Google Cloud (Mandiant). (2026, March 24). M-Trends 2026.cloud.google.com
3CrowdStrike. (2026, February 24). 2026 CrowdStrike Global Threat Report.crowdstrike.com
4IBM. (2025, July 30). IBM report: 13% of organizations reported breaches of AI models or applications, 97% of which reported lacking proper AI access controls.newsroom.ibm.com
5CyberArk. (2025, April 23). 2025 Identity Security Landscape (Business Wire).businesswire.com
6SailPoint. (2025, May). AI agents: the new attack surface.sailpoint.com
7Triedman, H., Jha, R., & Shmatikov, V. (2025). Multi-agent systems execute arbitrary malicious code. arXiv 2503.12188; COLM 2025.arxiv.org
8Lee, D., & Tiwari, M. (2024). Prompt infection: LLM-to-LLM prompt injection within multi-agent systems. arXiv 2410.07283.arxiv.org
9Gartner. (2026, April 28). Gartner identifies six steps to manage AI agent sprawl.gartner.com
10OWASP GenAI Security Project. (2025, December 9). OWASP Top 10 for Agentic Applications for 2026.genai.owasp.org
Rogue AI Agents · Part 2
Sources · 2 of 2
Platforms, regulators and guidance.
11MITRE. ATLAS data releases (v2026.09).github.com
12OpenTelemetry. Semantic conventions for generative AI agent spans.github.com
13AWS. (2025, October 13). Amazon Bedrock AgentCore is now generally available.aws.amazon.com
14Microsoft. Microsoft 365 roadmap: Agent 365.microsoft.com
15Monetary Authority of Singapore. Notice FSM-N07 on technology risk management, and Circular on financial institution incident reporting (December 2025).mas.gov.sg
16Monetary Authority of Singapore. (2026, October 7). MAS sets out supervisory expectations on responsible AI adoption by financial institutions.mas.gov.sg
17CMS. (2025, October). Cyber incident response and compliance with DORA.cms.law
18EU AI Act, Article 26: Obligations of deployers of high-risk AI systems.artificialintelligenceact.eu
19Cyber Security Agency of Singapore. (2026). Securing agentic AI: addendum to the Guidelines on Securing AI Systems.csa.gov.sg
20Crowell & Moring. (2026). American and allied cyber agencies issue first joint guidance on securing agentic AI.crowell.com
Rogue AI Agents · Part 2
Thank you
Follow the actions.
Read the full series at terencekok.com/blog/series/rogue-ai-agents/
Terence Kok and Maria SinghPart 3 turns to the controls: autonomy tiers, hard limits and kill switches.
