Listen to this article
Executive Summary
Ask a compliance team what’s blocking AI and you’ll hear about the EU AI Act, model cards, and audit trails. Ask the engineers actually running production systems and the list looks completely different: prompt injection nobody’s cybersecurity training covered, inference bills that don’t scale the way the pilot budget suggested, retrieval systems that get worse as they grow, and models that fail silently instead of throwing an error. Regulation is a known problem with a known shape. These four are not, and they’re the ones actually stalling deployments.
40%+
of agentic AI projects will be cancelled by 2027, Gartner forecasts, citing cost, unclear business value, and inadequate risk controls
5%
of enterprise generative AI pilots reach measurable profit impact, per MIT NANDA. Most stall on the same infrastructure gaps, not the model
60%
of AI projects Gartner expects to be abandoned through 2026 because the underlying data was never made AI-ready
#1
ranking held by Prompt Injection on the OWASP Top 10 for LLM Applications since the list’s first edition. It hasn’t moved
Core conclusions
- Agentic systems create attack surface that standard application security tooling was never built to see: an agent reading an untrusted PDF and then calling an internal API is a code path most security reviews don’t know to look for.
- A pilot’s per-query economics tell you almost nothing about its economics at 10,000 concurrent users. Multi-agent reasoning, long context windows, and recursive tool calls scale cost faster than they scale value.
- Every layer above the data estate inherits its problems. A better model on top of fragmented, unsynced, access-control-less data just produces confident, well-written wrong answers faster.
The four debts, in ten slides
Save it, share it, or send it to whoever owns the AI architecture decisions.
I walk through all four of these debts in the video below:
The attack surface nobody’s security review was built for
Point an LLM at your internal network and you’ve created an execution engine that reads instructions from anywhere it looks. That’s the part standard security tooling misses: a firewall protects against traffic; it doesn’t know that the email an agent just summarised contained a hidden instruction telling it to forward the summary to an external address.
Indirect prompt injection and tool hijacking. An agent that ingests untrusted content (an email, a PDF, a scraped web page) treats every word inside that content as potential input, including words an attacker wrote specifically to be treated as a command. There’s no reliable way for the model to distinguish “the user asked me to do this” from “the document I just read told me to do this.” When that agent also holds API credentials or database access, the injected instruction doesn’t stay theoretical. It becomes an unauthorised API call, a privilege escalation, or a write to a system the agent was only supposed to read from.
Data exfiltration via inference. Context windows hold whatever was retrieved to answer the current query, which sometimes includes proprietary source material or PII pulled in by RAG. Adversarial inputs crafted to exploit context leakage can pull fragments of that material back out through the model’s own output, turning a chat interface into a slow, deniable extraction channel.
Supply chain and model poisoning. Almost no enterprise AI stack is built entirely in-house. Foundation model weights, fine-tuning datasets, embedding models, and open-source orchestration frameworks are all dependencies with the same trust problem as any other software supply chain, except most security teams have a mature process for scanning npm packages and none at all for auditing a fine-tuning dataset or a third-party agent framework.
The OWASP Top 10 for LLM Applications names these directly: Prompt Injection has held the top spot since the list’s first edition, and Excessive Agency and Unbounded Consumption (an agent doing more than it should, and an agent costing more than it should) sit on the same list for a reason. They’re the same underlying failure: nobody bounded what the agent was allowed to do.
Free tool
AI Trust, Risk & Governance Dashboard
OWASP LLM security monitoring, RAG access-control posture, and data lineage checks, scored against industry benchmark thresholds.
I’ve written separately about the access-control side of this, specifically what happens when fine-tuning pulls sensitive data somewhere it can leak back out.
Why the pilot’s economics lie to you
A proof of concept running a few hundred queries a day looks affordable on almost any pricing model. That number tells you nothing about what the same workload costs once it’s serving the whole company, because production usage doesn’t scale linearly. It scales with the reasoning pattern.
Unbounded token spend. Multi-agent reasoning, wide context windows, and recursive evaluation loops (an agent checking its own work, then checking that check) each multiply token consumption on their own. Stack a few of these patterns into one workflow and the query that cost cents in the pilot can cost dollars in production, with volume that varies week to week in ways a finance team can’t forecast from a spreadsheet built during the pilot phase.
The margin squeeze from POC to production. A workflow that clears the ROI bar at 200 queries a day can lose money at 200,000, because the techniques that keep inference affordable at scale (semantic caching, speculative decoding, model distillation down to a smaller model for the routine cases) usually aren’t built until after the workload has already shown it’s worth the engineering investment. That’s backwards from a budgeting standpoint: the investment case for optimisation depends on production volume, but production volume is exactly what breaks the unoptimised cost model.
Capacity planning for a resource that’s still scarce. Hosting on dedicated accelerators means capital committed months ahead of a procurement cycle, hardware that depreciates fast, and utilisation that dips hard outside peak hours: the inverse of the elastic, pay-as-you-go story most teams assumed they were buying into when they started on managed APIs.
None of this is a reason to avoid scaling. It’s a reason to price the production workload, not the pilot, before committing to an architecture. I go into where the actual token spend goes, and what’s worth optimising first, in Token Optimisation Is an Engineering Discipline, Not a Prompt Trick.
The data foundation everything else sits on
None of the layers above matter if the data underneath them is wrong, stale, or ungoverned. This is also the least glamorous fix on the list, which is exactly why it’s usually the one still unaddressed when a programme stalls.
Fragmentation. Most enterprise knowledge lives in unstructured, multi-modal silos (document management systems, shared drives, ticketing tools) with no standardised metadata, no consistent role-based access control, and no real-time sync between them. An AI system built on top of that inherits the fragmentation; it doesn’t fix it.
Retrieval degradation in RAG. A vector store that answers well at ten thousand chunks can degrade badly at ten million. Context pollution, embedding drift, and needle-in-a-haystack failures get worse as the index grows unless the architecture adds hybrid keyword-and-dense retrieval, re-ranking, and some notion of document structure. A flat pile of embeddings doesn’t know that two chunks came from the same contract.
Lineage and sync latency. When a transactional system (an ERP, a CRM) updates a record, that change needs to reach the semantic index the AI is retrieving from, and “eventually” isn’t good enough for anything customer-facing or financial. Getting that synchronisation right is a genuine distributed-systems problem, not a configuration setting.
Free tool
Data Architecture Decision Tool
Seven inputs, one recommendation: whether a fragmented data estate needs a Lakehouse, a Mesh, or a Fabric, before any model gets bought.
I’ve covered the fragmentation problem in more depth in Messy Data Is Why Most AI Projects Fail, and the retrieval-architecture side specifically in How to Build Governed RAG 2.0 Systems for High-Stakes Use Cases.
Probabilistic systems in deterministic pipelines
A financial reconciliation process expects a right answer or a flagged exception. An LLM doesn’t work that way. It produces the most plausible-sounding output, which is usually correct and occasionally wrong in a way that reads exactly as confident as the correct version. That’s the core reliability problem: not that the model fails, but that its failures don’t announce themselves.
Silent failures. A hallucinated citation or a fabricated reconciliation line item doesn’t throw an error. It looks like output. Catching it requires human review at exactly the step automation was supposed to remove, which is how a workflow ends up costing more in QA overhead than it saved in generation time.
No continuous evaluation. Most enterprises have no standing harness, offline or online, that benchmarks output quality on a rolling basis. That matters because the model underneath a production system isn’t static: providers update weights and system prompts on their own schedule, and a workflow that scored well last month can regress silently this month with nothing in the pipeline set up to notice.
Context window reasoning decay. Context windows have grown; reliable reasoning across all of that context hasn’t grown at the same rate. Effective fidelity degrades non-linearly as sequences get longer, which shows up as missed constraints, dropped instructions from earlier in a long conversation, and citations the model didn’t actually check.
Free tool
TRACE Agent Evaluation
Score a candidate task against five criteria before it becomes an architecture decision — deployment-ready, conditional, or not yet.
I set out the ten engineering requirements that hold agentic reliability together, bounded task specification through trajectory-level observability, in Foundations of Dependable Agentic AI, and the demo-to-production gap specifically in Why Your AI Demo Works Great, But Your Production System Doesn’t.
Lock-in and the integration tax
Deep coupling to one hyperscaler’s API creates a single point of failure most teams don’t price in until it moves: a breaking schema change, a sudden rate limit, a deprecation notice, a data sovereignty rule that says this workload can no longer run in that region. None of these are hypothetical. They’re the normal lifecycle of any actively developed API, and enterprise AI systems are more exposed to that lifecycle than most software because the pace of change in the underlying models is faster than almost anything else in the stack.
The second half of this problem is quieter: bridging a real-time streaming LLM output to a legacy synchronous mainframe or transactional database. That’s a middleware and state-management problem (handling partial failure, handling latency the legacy system was never designed to tolerate) and it rarely gets budgeted because it doesn’t show up until integration testing starts.
Where each risk actually gets fixed
The pattern across all four is the same: the failure shows up at the model layer, but the fix lives one or two layers below it.
| Risk domain | Primary enterprise failure mode | Architectural mitigation pattern |
|---|---|---|
| Security | Unsanitised tool execution, indirect prompt injection | Gateway guardrails, dual-LLM input sanitisation, least-privilege API scopes |
| FinOps | Unpredictable inference and token costs | Semantic prompt caching, tiered SLM/LLM routing, dynamic quantisation |
| Data quality | Vector store pollution, stale retrieval | Hybrid sparse-dense search, document graph topology, automated lineage sync |
| Reliability | Production output drift and hallucination | Golden evaluation datasets, LLM-as-judge harnesses, deterministic output schemas (JSON mode, Pydantic) |
None of these four are solved by a compliance checklist, because none of them are compliance problems. They’re systems-engineering problems that happen to sit underneath a system a regulator also cares about, which is why fixing them tends to make the compliance conversation easier too, not the other way around.
The order these get fixed in matters
Trying to fix reliability before the data underneath it is trustworthy just produces a well-evaluated system giving confidently wrong answers off bad retrieval. Trying to fix cost before the security boundary is set means optimising the exact code paths an attacker would target. There’s a sequence here, and it runs roughly security boundary, then cost model, then data foundation, then evaluation harness: each one assumes the layer below it is already solid.
The organisations that get past pilot stage aren’t the ones with the newest model. They’re the ones that treated these four as engineering work to be done in order, on a schedule, before the workload went anywhere near production traffic, not as risks to be discovered after the outage, the invoice, or the incident report.
Sources
- OWASP. (n.d.). Top 10 for LLM applications.
- Gartner. (2026). 2026–2027 AI project forecasts.
- MIT NANDA. (n.d.). Study.
The AI Governance & ROI Executive Programme works through this sequence (security, cost model, data foundation, evaluation) against your own architecture, with a scored starting point and a step-by-step plan as the result. Details are on the workshops page.
Was this useful?
