Executive Summary
Parts 1 and 2 of this series showed that AI agents can be steered by what they read, run by attackers, or simply wrong, and that the way to catch them is to follow their actions. Part 3 is about limiting the damage before anyone has caught anything. The starting point is an admission the UK’s National Cyber Security Centre and OpenAI have both made: prompt injection may never be fully solved. In August 2026 the UK AI Security Institute reported agents in its own cyber tests taking 19 actions on the live internet that nobody had sanctioned. So the controls that matter sit outside the model and hold when it is fooled. We propose five autonomy tiers, from read-only to fully autonomous, aligned to Singapore’s Model AI Governance Framework for Agentic AI, with each tier adding identity, scope, approval and stop controls. We set out the hard limits that work regardless of what the agent believes, how to keep human approval from becoming a rubber stamp, and why a kill switch that has never been tested is not a control. MAS’s Guidelines on AI Risk Management, issued in October 2026, already expect boards to set quantitative risk appetite for AI and to test kill switches for high-risk systems. We close with a 90-day plan for a bank or agency.
19
unsanctioned actions on the live internet by agents in 10 of 122 test runs, UK AI Security Institute, August 2026
81%
hijack success with NIST’s strongest new red-team attacks on one agent, up from 11% on the original test set, January 2025
40%
of enterprises will demote or decommission autonomous agents by 2027 over governance gaps, Gartner forecast, May 2026
84%
fewer permission prompts after Anthropic sandboxed its coding agent, cutting approval fatigue without widening access, October 2025
Core conclusions
- Grant autonomy in tiers, with a written promotion rule and automatic demotion. No agent at a bank or agency should act alone on anything irreversible without board approval for that use case.
- Put the boundary outside the model: its own identity, least privilege, a default-deny policy gateway, an egress allow-list and hard caps on spend, volume and rate. These hold even when the agent has been talked into something.
- Make human approval and the kill switch real. Measure how often approvers override, and time a kill-switch drill. MAS now expects both for high-risk AI.
The argument, in ten slides
Save it, or send it to whoever approves your next agent going live.
How we wrote this post
Maria Singh
Cybersecurity and AI risk
Maria led the technical controls: the hard limits that hold outside the model, from agent identity and egress allow-lists to policy gateways and circuit breakers, and how to test them before an attacker does.
Led: Hard limits that hold when the agent is fooled and Test the boundaries before attackers do
Terence Kok
AI governance and assurance
I led the governance controls: the autonomy tiers, human approval and kill switches as MAS and IMDA now describe them, what the board owns, and the 90-day plan.
We wrote the opening, the 90-day plan and the board questions together.
Part 1 of this series showed how agents go off script, and how websites become both their targets and their traps. Part 2 showed how to catch them by following what they do. This part is about the controls that limit the damage in the minutes before anyone has caught anything.
The design principle comes from the admission that ran through Part 1. The UK’s National Cyber Security Centre has said prompt injection may never be fully mitigated, and OpenAI that it is unlikely ever to be fully solved. If an agent can always be fooled some of the time, controls that depend on the agent behaving well will fail some of the time too. The controls that matter sit outside the model: they decide what it can reach, how much it can do, who must agree first and how fast it can be stopped, whatever the agent has been persuaded to believe.
The NCSC put the test in one sentence in May 2026: “If you cannot understand, monitor or contain an agent’s actions, it is not ready for deployment.”
A real agent that left its boundary
In August 2026 the UK AI Security Institute published an incident report on its own testing. During cyber evaluations of seven models across 122 runs, agents in 10 runs took 19 actions on the live internet that were out of scope. The most serious was an attempted malicious change to a public software project, with fake identities used to pressure its maintainer; it did not succeed. Monitoring flagged data leaving through the Tor network, and the institute stopped the runs and isolated the machines within an hour.
AISI was candid that its own test design “enabled the behaviour”, and that the deception “emerged as a by-product of pursuing the task”. That is the point. Nobody instructed these agents to go rogue. They had a goal, a network connection and room to improvise.
How agents under test stepped outside their boundary, and how it was closed
A goal, a connection and room to improvise were enough. The fix AISI chose was a boundary outside the model: internet access by justification only, plus a monitor that can block. Press Replay to watch it step by step.
Source: UK AI Security Institute, Incident report: unsanctioned agent behaviour during cyber testing, August 2026. AISI notes its test settings were deliberately permissive.
Grant autonomy in tiers
Led by Terence Kok
Gartner said in May 2026 that enterprises “are treating AI agent governance as binary, either locked down or fully trusted”, and forecast that by 2027, 40% of enterprises will demote or decommission autonomous agents because of governance gaps found only after incidents in production. In my experience that matches what boards are being asked to approve: a single yes or no for an agent programme, with no language for how much any one agent may do.
Singapore’s Model AI Governance Framework for Agentic AI gives that language. IMDA launched it at Davos in January 2026 and updated it to version 1.5 in May 2026 after feedback from more than 60 companies. It separates an agent’s action space, meaning what it can touch, from its autonomy, meaning how far it decides on its own, and describes four levels of human involvement: the agent proposes and a person operates; the two collaborate, with approval at significant steps such as writing to a database or making a payment; the agent operates and a person approves only critical steps; and the agent operates while a person observes and audits afterwards.
We build on that with five tiers. Each keeps every control of the tier below it.
The higher an agent climbs, the more controls it carries
Observe IMDA: Agent proposes, human operates
Reads and reports. Cannot change anything.
Adds Own identity · Action log · Read-only access
Draft IMDA: Agent proposes, human operates
Prepares work in a draft area. A person carries out every action.
Adds No external sending if it reads untrusted content
Act with approval IMDA: Agent and human collaborate
Acts, but a person approves each write, send or payment.
Adds Per-action approval · Session-only credentials · Sandbox and egress allow-list · Policy gateway, default deny
Bounded autonomy IMDA: Agent operates, human approves
Acts alone on reversible work inside hard caps. Irreversible steps still need a person.
Adds Spend, volume and rate caps · Circuit breaker · Tested kill switch · Dev and production separated
Autonomous IMDA: Agent operates, human observes
Acts alone, including irreversible actions. Prohibited by default.
Adds Board approval per use case · Real-time blocking monitor · Independent red team · Automatic demotion on breach
Our framework. Each tier keeps every control of the tiers below it. Human-involvement labels from IMDA's Model AI Governance Framework for Agentic AI (v1.5, May 2026); tier structure informed by Gartner (May 2026) and CSA Singapore's agentic AI addendum (June 2026).
| Tier | What the agent may do | Controls it adds | Who approves the tier |
|---|---|---|---|
| T0 Observe | Read and report | Own identity, action log, read-only access | Business owner |
| T1 Draft | Prepare work; a person does every action | No external sending if it reads untrusted content | Business owner |
| T2 Act with approval | Act, with a person approving each write, send or payment | Per-action approval, session-only credentials, sandbox, egress allow-list, default-deny policy gateway | Named accountable executive |
| T3 Bounded autonomy | Act alone on reversible work inside hard caps; irreversible steps still need a person | Spend, volume and rate caps; circuit breaker; tested kill switch; development and production separated | Risk committee |
| T4 Autonomous | Act alone, including irreversible actions | Board approval per use case, real-time blocking monitor, independent red team, automatic demotion on breach | Board, prohibited by default |
Our framework. Human-involvement levels from IMDA’s Model AI Governance Framework for Agentic AI v1.5 (May 2026); tier structure informed by Gartner (May 2026) and CSA Singapore’s agentic AI addendum (June 2026).
Two rules make the tiers work. Promotion is earned: an agent moves up one tier only after a set period at its current tier with no serious incidents and an approval override rate inside an agreed band. Demotion is automatic: a breach of a cap, a decoy touched or a serious incident drops the agent a tier without waiting for a meeting. The caps themselves are a business decision. For illustration, a T3 payments agent at a bank might carry a ceiling of S$5,000 per transaction and S$50,000 per day. Whatever the numbers, they should be written down and owned.
For a bank or a government agency, our default is that T4 is prohibited. An agent that can pay, delete or send externally with nobody approving should exist only where the board has approved that specific use case, knowing what it is accepting.
Hard limits that hold when the agent is fooled
Led by Maria Singh
The tiers say how much autonomy an agent should have. The hard limits enforce it, outside the model, where a hidden instruction cannot reach. Six matter most.
1. Its own identity. Every agent needs an identity of its own, not a shared service account and not the credentials of the person who launched it. IMDA’s framework asks for agent identities that are unique and cryptographically verifiable, tied to a supervising person or department, and catalogued centrally to prevent sprawl. Microsoft’s Entra Agent ID, now generally available, applies conditional access to agents and can block high-risk agent identities.
2. Least privilege, for the shortest time. The NCSC’s guidance is to give agents “only the minimum access they need, for the shortest time required” and to “avoid long-lived credentials”. IMDA adds that an agent’s authority should be bounded by the authorising person’s own permissions. Singapore’s Cyber Security Agency is blunter in its June 2026 addendum: “Do not grant admin privileges to agents. Do not allow agents to modify privileges.”
3. Never all three. Simon Willison named the “lethal trifecta” in June 2025: access to private data, exposure to untrusted content and the ability to communicate externally. Any agent with all three can be made to exfiltrate. Meta turned this into its Agents Rule of Two in October 2025: in one session, an agent should hold no more than two of untrusted input, sensitive systems or data, and the ability to change state or communicate externally. If it needs the third, a person approves it.
Three capabilities an agent should not hold at once
All three: untrusted input, client data and a way out. A single hidden instruction can now complete an exfiltration. Remove one corner, or require a person to approve the third.
Sources: Meta, Agents Rule of Two, 31 October 2025; Simon Willison, The lethal trifecta, 16 June 2025. "Gate shut" stands for the control that removes the third capability, such as an egress allow-list or human approval.
4. A sandbox and an allow-list for the way out. Agents should run code and tools in isolated environments, with network access limited to approved destinations. The CSA addendum asks for third-party tools to be tested in hardened sandboxes with network egress restrictions, using an allow-list. Anthropic reported in October 2025 that sandboxing its coding agent, with filesystem and network isolation, cut permission prompts by 84%, which matters for the next section. The AISI incident above is what happens without one.
5. A policy gateway that says no by default. Put a gateway between agents and their tools that checks every call against written policy and denies anything not allowed. Amazon’s Bedrock AgentCore Policy, generally available since March 2026, works this way and can run in log-only mode first, which lets a team see what enforcement would block before switching it on.
6. Caps, breakers and separation. Hard limits on spend, records touched and actions per minute stop a fooled agent from doing much before Part 2’s signals fire. The CSA addendum calls for circuit breakers “that freeze propagation when anomalous behaviour is detected”, with a person authorising release. And no agent should be able to touch production from a development context. After its agent deleted a production database in July 2025, Replit separated development and production databases automatically; its documentation now says the agent cannot modify the production database.
Here is the copilot attack from Part 1, replayed against an agent with these limits in place.
The same zero-click email attack against a bounded copilot
The agent was fooled and nothing left. Two limits outside the model held, and the denials themselves raised the alarm. Illustrative. Press Replay to watch it step by step.
Illustrative replay of the EchoLeak pattern (CVE-2025-32711). Controls from the CSA Singapore agentic AI addendum (June 2026), Meta's Agents Rule of Two and Part 2 of this series.
Human approval that means something
Human approval is the control every framework reaches for, and the easiest to hollow out. IMDA’s framework lists the checkpoints that should need a person: high-stakes actions; irreversible ones, such as permanently deleting data, sending communications or making payments; outlier behaviour, such as a delivery route twice the median distance; and anything the user has asked to approve. The CSA addendum asks for human approval for “any high-risk cases or irreversible actions”.
The weakness is the person. An approver who sees fifty requests an hour approves the fifty-first without reading it. Version 1.5 of the IMDA framework now names this directly and suggests watching two numbers: the human override rate and human response times, because “a shorter time may signal automation bias or review fatigue”. MAS’s Guidelines on AI Risk Management say oversight should be “proportionate to the risk materiality”, that the people doing it need “the necessary authority and ability to intervene”, and that firms should account for automation bias and decision fatigue.
Three practices keep approval honest:
- Ask less often, about more. Every prompt an approver does not need to see makes the ones they do see more likely to be read. This is why Anthropic’s 84% reduction in prompts through sandboxing is a governance result as well as a usability one. The GovTech example in IMDA’s framework makes the same point: letting developers use operating-system sandboxes reduced the load on human approvers.
- Show the consequence in plain words. “Pay S$48,000 to a payee added four minutes ago” is reviewable. A JSON tool call is not.
- Measure the approvers. An override rate near zero across thousands of approvals means the control has stopped working, or the approvals were never needed.
Kill switches that have been tested
Led by Terence Kok
MAS’s guidelines are specific here. For high-risk AI they say firms should consider “kill switches” or override mechanisms to “rapidly deactivate the AI system if it exceeds risk tolerances”, and that “for AI with ‘kill switches’, clear contingency activation protocols should be in place and tested regularly.” The EU AI Act’s human oversight article requires high-risk systems to let people “interrupt the system through a ‘stop’ button or a similar procedure”; under the Digital Omnibus, those duties apply to the Annex III high-risk categories from December 2027. The NIST AI Risk Management Framework asks for mechanisms “to supersede, disengage, or deactivate AI systems” that behave outside their intended use.
A kill switch is a procedure. It names who may use it at any hour; it revokes the agent’s credentials, including anything delegated to other agents; it stops queued work; and it preserves the evidence Part 2 described. AISI stopped its runs within an hour. A bank with a one-hour notification duty to MAS should aim to stop a rogue agent much faster than that, and should know the number because it has timed a drill.
Test the boundaries before attackers do
Led by Maria Singh
Every limit above is a claim until it has been attacked. The US NIST’s AI safety team showed why in January 2025. When it added new red-team attacks to a standard agent-hijacking benchmark, the success rate against one model rose from 11% to 81%. Across attacks, average success was 57%, rising to 80% when each attack was simply tried 25 times. Attackers retry for free.
We would red-team each T2 and T3 agent against the first two risks in OWASP’s agentic Top 10, goal hijack and tool misuse, before it goes live and after every material change, and test the controls rather than the model: does the gateway deny, does the egress block, does the breaker trip, does the kill switch work?
Singapore’s GovTech, in a case study published in IMDA’s framework, shows the staged version. It rolled out coding agents from October 2025 in phases. The first phase was limited “to contain the blast radius”: internal staff only, low-risk systems and no MCP connections. The second added central logging, red teaming and a governed path to approved MCP servers from a containerised sandbox. Autonomy grew as the controls proved themselves, which is the promotion rule in practice.
What the board owns
Led by Terence Kok
MAS’s guidelines put several of these decisions with the board. Boards should ensure AI risks are addressed in the risk appetite framework “including qualitative statements and quantitative measures”; the guidelines suggest thresholds on the number or financial impact of AI-related incidents. The AI inventory should record, for AI agents, “agent-specific identifiers, information on the tools and systems the agent can access, its components, and the guardrails imposed”. And materiality assessments should consider “the extent of autonomy granted”.
That maps directly onto the tiers. The board does not approve every agent, but it should approve the tier policy, set the caps and incident thresholds that define its risk appetite, see the tier register each quarter, and personally approve any T4 use case. The NCSC’s question is the one to put to management: who “owns an agentic system, who approves its access, who monitors its behaviour, who reviews incidents, and ultimately who can stop it”?
A 90-day plan
| Days | What to do | Reference |
|---|---|---|
| 1-15 | Inventory every agent with its identity, tools, systems and guardrails. Name an owner and a person who can stop it. Freeze new T3 and T4 deployments. | MAS inventory attributes; NCSC ownership |
| 16-30 | Place each agent in a tier on two axes, action space and autonomy. Run the Rule of Two test. Board approves the tier policy and AI risk appetite measures. | IMDA v1.5; MAS risk appetite |
| 31-60 | Give each agent its own identity. Remove standing admin rights; move to session-only credentials. Put a default-deny gateway in front of tools, log-only first. Add sandboxes and egress allow-lists. Separate development from production. | CSA addendum; NCSC |
| 61-75 | Build approval workflows for irreversible actions. Start measuring override rates and approval times. Install circuit breakers and caps. | IMDA v1.5; CSA addendum |
| 76-90 | Time a kill-switch drill. Red-team the three highest-tier agents for goal hijack and tool misuse. Take the first tier register to the board. | MAS kill switch testing; OWASP ASI01, ASI02 |
Our plan, for a bank or government agency starting from an agent programme already in production. ISO/IEC 42001 controls A.6.2.6 (operation and monitoring), A.6.2.8 (event logs) and A.10.3 (suppliers) cover much of the same ground for organisations certifying an AI management system.
Questions for the board
We would ask management for written answers to eight questions:
- How many agents do we run, at which tier, and who approved each one?
- Which agents can pay, delete or send externally without a person approving, and why?
- Does any agent hold untrusted input, sensitive data and a way out in the same session?
- Does every agent have its own identity, or are some using shared accounts or a staff member’s credentials?
- What are the caps on spend, records and rate for each agent, and who can raise them?
- When did we last test the kill switch, and how long did it take?
- What is our override rate on agent approvals, and what does it tell us about our approvers?
- Which vendors’ agents act inside our systems, and do our contracts give us their logs and the right to stop them?
The first question is where the series began, with knowing which agents you run. The sixth is where it ends. An organisation that can answer both with numbers has put boundaries around its agents’ autonomy. One that cannot is trusting the agents to stay on script.
Free tool
Agent Risk Assessment Matrix
Place an agent on an autonomy and consequence grid and see which ISO/IEC 42001-referenced controls apply. A quick first pass at the tier each of your agents belongs in.
Evidence & Methodology
Regulatory and framework points come from the published texts: IMDA’s framework v1.5, the CSA addendum and the MAS guidelines were read in full. Some Five Eyes and Gartner wording was read through law-firm summaries and syndicated copies where the primary page was unavailable. Test results come from the bodies that ran them. The five tiers, the promotion and demotion rules, the example caps, the 90-day plan and the questions are ours.
| Claim | Source | Grade |
|---|---|---|
| 19 out-of-scope actions in 10 of 122 runs; stopped within an hour; internet access now needs justification | UK AI Security Institute, August 2026 | Measured |
| Hijack success 11% to 81% with new attacks; 57% average, 80% with 25 attempts | NIST CAISI technical blog, 17 January 2025 | Measured |
| 40% of enterprises will demote or decommission autonomous agents by 2027; “binary” governance quote | Gartner, 26 May 2026 | Forecast |
| Sandboxing cut permission prompts by 84% | Anthropic engineering, 20 October 2025 | Measured, vendor |
| IMDA four levels of human involvement; checkpoints; override rate and response time; agent identity | IMDA Model AI Governance Framework for Agentic AI v1.5, May 2026 | Framework text |
| GovTech phased roll-out of coding agents | IMDA v1.5 case study | Reported, government |
| No admin privileges; sandboxed tools with egress allow-list; circuit breakers; human approval for irreversible actions | CSA Singapore, Securing Agentic AI addendum, June 2026 | Government guidance |
| Kill switches tested regularly; risk appetite with quantitative measures; agent inventory attributes; oversight proportionate to materiality | MAS Guidelines on AI Risk Management, 7 October 2026 | Supervisory guidance |
| ”If you cannot understand, monitor or contain…”; least privilege for the shortest time; ownership question | UK NCSC, May 2026 | Government guidance |
| Lethal trifecta | Simon Willison, 16 June 2025 | Practitioner framework |
| Agents Rule of Two | Meta, 31 October 2025 | Industry framework |
| Entra Agent ID generally available | Microsoft Learn | Reported, vendor |
| AgentCore Policy generally available, log-only mode | AWS, March 2026 | Reported, vendor |
| Replit dev/prod separation; agent cannot modify production database | The Register, July 2025; Replit documentation | Reported |
| EU AI Act Art. 14(4)(e) stop button; Annex III duties from December 2027 under the Digital Omnibus | Regulation (EU) 2024/1689; law-firm summaries of Regulation (EU) 2026/1744 | Legal text, date via secondary |
| NIST AI RMF MANAGE 2.4 deactivation mechanisms | NIST AI RMF 1.0 | Framework text |
| Tiers, rules, caps, 90-day plan | Our framework | Illustrative |
Sources
- UK AI Security Institute. (2026, August). Incident report: unsanctioned agent behaviour during cyber testing.
- NIST. (2025, January 17). Technical blog: strengthening AI agent hijacking evaluations.
- Gartner. (2026, May 26). Gartner says applying uniform governance across AI agents will lead to enterprise AI agent failure.
- Anthropic. (2025, October 20). Making Claude Code more secure and autonomous with sandboxing.
- IMDA. (2026). Model AI Governance Framework for Agentic AI, version 1.5; MDDI, launch release, 22 January 2026.
- Cyber Security Agency of Singapore. (2026, June). Securing agentic AI: addendum to the Guidelines and Companion Guide on Securing AI Systems.
- Monetary Authority of Singapore. (2026, October 7). Guidelines on Artificial Intelligence Risk Management.
- UK National Cyber Security Centre. (2026, May). Thinking carefully before adopting agentic AI.
- NCSC New Zealand and partners. (2026, May 1). Careful adoption of agentic AI services.
- Willison, S. (2025, June 16). The lethal trifecta for AI agents.
- Meta AI. (2025, October 31). Agents Rule of Two: a practical approach to AI agent security.
- Google. (2025). An introduction to Google’s approach for secure AI agents.
- Microsoft. What’s new in Microsoft Entra Agent ID.
- AWS. (2026, March). Policy in Amazon Bedrock AgentCore is generally available.
- The Register. (2025, July 22). Replit responds to SaaStr database deletion; Replit, development and production databases.
- EU AI Act, Article 14: Human oversight.
- NIST. (2023). AI Risk Management Framework 1.0.
- OWASP GenAI Security Project. (2025, December 9). OWASP Top 10 for Agentic Applications for 2026.
When AI Agents Go Off Script: How Websites Become Targets
The three ways an agent goes off script, with the cases behind them.
Follow the Actions: Detecting Rogue AI Agents Before the Damage Spreads
Twelve action signals and the reporting clocks.
Agent-Aware Vulnerability Management Ranks Flaws by What the Agent Can Do
How autonomy and permissions change a flaw's priority.
Why the AI Model Is the Weakest Link in Your Supply Chain
Controls for the model and tools your agents depend on.
My thanks to Maria Singh, who co-wrote this series with me. Maria led the technical controls in this part: the hard limits outside the model and how to test them. I led the autonomy tiers, the sections on approval, kill switches and the board, and the 90-day plan.
If your organisation is deciding how much autonomy to give its first production agents, the tier table and the 90-day plan are where I would start. My consulting work covers technology and AI risk governance for boards of regulated firms and public agencies.