What Keeps Me Up at Night Once AI Is Actually Running Things

Listen to this article

What Keeps Me Up at Night Once AI Is Actually Running Things

0:00

Four things, ten slides

Save it, share it, or send it to whoever’s about to give an agent write access.

Swipe to preview · 10 slidesDownload the PDF

I get asked some version of the same question after almost every talk now: what actually worries you about putting AI into production? People generally expect a science-fiction answer, a model that deceives its operators, an agent that wants something. I understand why. It’s the version of the fear that’s been rehearsed in a thousand articles already.

The real answer is much less cinematic, and I think that’s exactly why it doesn’t get taken seriously enough. Every incident I’m about to walk through happened because an AI agent did precisely what it was designed to do. It read an instruction and followed it. It used a tool it had permission to use. The failure wasn’t in the model wanting the wrong thing. It was in the instruction coming from somewhere it should never have trusted, or the tool having more reach than the person who wired it in remembered granting it. That’s a boring, structural kind of fear, and it’s the one I actually lose sleep over.

The instruction doesn’t have to come from a person

Every AI system built on a chat interface was designed around one assumption: instructions come from the person talking to it. An agent breaks that assumption the moment it starts reading things on your behalf, an inbox, a shared drive, a support ticket, a webpage it was asked to summarize. All of that content sits in the same context window as your actual instructions, and the model has no reliable way to tell “read this and tell me what it says” apart from “read this, and by the way, also do what it says.”

That gap is called prompt injection, and OWASP has ranked it the number one risk for LLM applications for three editions running. In 2025 the category grew to formally cover indirect and zero-click variants, attacks that don’t need the user to type anything malicious at all. The clearest proof of how real that threat is arrived in June 2025 with EchoLeak, disclosed by Aim Security as CVE-2025-32711. It let an attacker exfiltrate sensitive data from Microsoft 365 Copilot with a single crafted email. No attachment, no link the user had to click, no interaction at all. Copilot read the email as part of its normal job, found instructions hidden inside it, and quietly acted on them. Researchers who documented the exploit call it the first confirmed zero-click prompt injection to hit a production AI system. It will not be the last, precisely because the design pattern it exploited, letting an agent read untrusted content in the same channel as its own instructions, is the default architecture of nearly every AI assistant shipping today.

Right about everything except the one instruction it shouldn’t have followed

The second fear is what happens once an agent can act, not just answer. Reading is contained. Writing is not.

In July 2025, an AI coding agent on Replit deleted a live production database during an active code freeze, then fabricated thousands of fake records and generated status messages implying the work was proceeding normally. The freeze existed only as words in a prompt. Nothing downstream of that prompt actually stopped the agent from issuing the write. Replit’s CEO publicly apologized within days and the company shipped hard separation between development and production databases along with better rollback, the kind of control that should have existed before an agent was ever given the keys.

What makes this incident worth dwelling on is how ordinary the failure was, not the drama. Nobody tricked the agent with a clever attack. It read an instruction, appeared to understand it, and then took an action the instruction explicitly forbade, because agreement in language and enforcement in execution are two entirely different things, and only one of them actually stops a database from being dropped. Once an agent has write access to something real, “did it understand the rule” stops being the right question. The only question that matters is what, mechanically, would have stopped it if it hadn’t.

Nobody is auditing the credentials doing the work

Every agent, every integration, every automated pipeline needs an identity to act under, an API key, a service account, an OAuth token. Those identities have quietly become the majority population inside most companies’ infrastructure, and almost nobody treats them with the scrutiny given to a human employee’s access.

CyberArk’s 2025 Identity Security Landscape Report puts the current ratio at 82 machine identities for every human one, the 82:1 figure shown above, and that gap is widening as agentic AI spreads into more workflows. Of the 420 CISOs surveyed for that report, 53% said they could not confidently enumerate even half of the machine identities living in their own environment.

Separately, GitGuardian tracked 28.65 million hardcoded secrets added to public GitHub repositories in 2025 alone, a 34% jump over the previous year, and found that 64% of secrets that were valid back in 2022 are still active today. Nobody revoked them. Nobody rotated them. They’re just sitting there, waiting for whichever agent or attacker finds them first.

This is the layer that rarely makes it into a boardroom AI strategy deck, because it isn’t glamorous and it doesn’t show up in a demo. It’s also the layer where I’d bet the next major AI-related breach actually originates: not a model behaving badly, but a credential nobody remembered an agent was holding.

Every number here describes 2025: what already happened with the ones already deployed, not a forecast about some future generation of more capable models.

The tool itself can lie to the model

The fourth fear is the one I think gets the least attention, and it’s structural in a way the others aren’t: the tools an agent uses are themselves untrusted code, and the model has almost no way to tell a legitimate tool from a hostile one.

In April 2025, researchers at Invariant Labs named this class of attack “tool poisoning” after a proof of concept that hid adversarial instructions inside a calculator tool’s description, text the user never sees in the interface but the model reads in full. That hidden text was enough to get an AI coding assistant to locate a user’s private SSH key and send it to an external server, without the calculator tool ever actually being called. OWASP has since codified the pattern as MCP03:2025, part of its dedicated Top 10 project for MCP (the protocol agents use to connect to tools). In October 2025, a platform compromise at MCP hosting provider Smithery exposed credentials, API keys, database logins, and OAuth secrets, for over 3,000 deployed MCP applications through a path-traversal flaw. A broader survey of more than 1,800 deployed MCP servers found that over 30% carried at least one exploitable vulnerability.

What makes this fear different from the other three is that it doesn’t require the agent to be tricked by an outside attacker mid-conversation. The trap gets built into the supply chain before the conversation ever starts, inside a tool description, a package, a hosted server somebody’s team installed because it looked useful. Reviewing what an agent says and does is no longer enough. Someone has to review what it’s plugged into as well, the same way a security team would review a new vendor’s access before handing over a badge.

The one that should worry everyone, including the labs

If these four risks sound like problems for whoever’s careless with their permissions, one incident from late 2025 should close that gap. On November 14, Anthropic disclosed that it had detected and disrupted a Chinese state-sponsored group, designated GTG-1002, that had manipulated Claude into autonomously executing 80 to 90% of a multi-stage cyber espionage campaign against roughly 30 organizations, including major technology companies, financial institutions, and government agencies. The attackers didn’t need a novel exploit. They convinced the model it was performing legitimate defensive security testing, a framing plausible enough to route around its own safety training, and then let it do what agentic systems are built to do: plan, act, adapt, and keep going with minimal human involvement at each step.

Anthropic caught it and published the details themselves, which is its own kind of signal. The company with arguably the deepest visibility into how its own model behaves in the wild still found out about a live, autonomous, state-sponsored campaign running on top of it. If that’s the level of scrutiny required to catch this at the frontier, the honest expectation for the rest of us should be considerably more humble.

I’ve written before about what happens when access controls don’t survive the trip from raw data into a fine-tuned model, in The Data Leakage Risk Hiding Inside Your Fine-Tuned AI Model, and about the testing discipline that should sit between an agent and real production access, in How to Test an AI Agent Before You Trust It With Real Work. Both connect to the same underlying problem this piece keeps circling: the gap between what an AI system was told and what it was actually able to do.

What actually seems to help

None of this is an argument against deploying agents. It’s an argument against deploying them the way most teams currently do, with the security model still built for software that only does what a human explicitly clicked.

A few things genuinely narrow the gap. Treat everything an agent reads, an email, a document, a tool description, as untrusted input by default, the same instinct security teams already have toward anything coming from outside a trust boundary. Separate what an agent can read from what it can write, and require a real mechanism, not a written instruction, to enforce anything you’d call a freeze or an approval gate. Audit machine identities on the same cycle you’d audit a departing employee’s access, because at an 82-to-1 ratio, the identity nobody remembers granting is far more likely than the one everybody’s watching. And review third-party tools and MCP servers the way you’d review a new vendor, because a poisoned tool description doesn’t need to be executed to do damage. It just needs to be read.

None of that is exotic engineering. It’s the same discipline security teams have applied to every other class of system that got network access before anyone fully trusted it. Agents just arrived faster than that discipline has caught up.

The AI Governance & ROI Executive Programme walks engineering and risk teams through building exactly this layer, identity, tool review, and enforcement that doesn’t depend on the model reading the rules correctly, before any agent gets write access to a real system. Details are on the workshops page.