Listen to this article
Jump to a section
In this article
Executive Summary
Agentic systems differ from single-shot inference in one crucial way: the model picks actions, sees what happens, and tries again. That changes where reliability comes from. It depends less on how capable the model is, and much more on the engineering built around it. Here are the ten elements that decide whether an agentic deployment holds up in real production, especially in regulated or safety-relevant settings.
Core conclusions
- A task only belongs to an autonomous agent if code can check whether it succeeded, and the agent runs against a hard step and cost budget. Everything else is decision support, not automation.
- The controls that matter most sit outside the model. They include narrow, validated tool schemas; least-privilege authority scoped to each task; an absolute boundary at OT and safety-instrumented systems; and treating every tool result as untrusted input by default.
- Assurance needs a record. That means logging every step of the trajectory, testing against known-correct trajectories, and using deterministic code wherever determinism is available.
The ten foundations, in ten slides
Save it, share it, or hand it to whoever is about to give an agent write access.
I have written before about the engineering checklist for shipping agentic systems to production once the architecture diagram is settled. This one sits a level below that. It covers the basic properties a system needs before “production” is even a fair word for it.
Foundations of dependable agentic AI
1. Bounded task specification
An agent needs a test of success that a machine can check, and an explicit termination condition. If code cannot check the objective, the task is a poor fit for autonomous execution. Restructure it as decision support instead. In practice this takes the form of a task contract, stated up front. It sets the input schema, the side effects allowed, the acceptance criteria, the maximum step budget, and the escalation trigger. A missing termination condition is the single most common cause of runaway loops and unbounded cost.
2. Context as an engineered resource
Model performance degrades non-linearly as the context fills with low-signal material. Good designs treat the context window as a budget to manage. Retrieval returns the least evidence that is enough. State lives in a durable store, rather than piling up as conversation history. Earlier steps are compacted from time to time into structured summaries. Retrieved content carries explicit provenance tags, so the agent can tell instruction from data. Sub-agent delegation is mainly a way to isolate context, not an organisational metaphor.
3. Tool surfaces designed for model consumption
Tool definitions are the main control surface. These features measurably cut failure rates:
- Narrow parameter schemas with enumerated values, validated before execution.
- Idempotency keys on all write operations, so retries after timeout do not duplicate effects.
- Error returns that state the corrective action.
- Token-efficient responses, with pagination and field selection, so a single call cannot exhaust the context.
- Consolidation of related operations into fewer, higher-level tools. A raw API surface of eighty endpoints reliably gives worse selection accuracy than fifteen task-shaped tools.
4. Closed-loop verification
Agentic systems that improve with each iteration share one property: they can reach a deterministic verifier. Compilers, unit tests, schema validators, physics or hydraulic simulators, reconciliation checks against a system of record, and constraint solvers all give the model a ground-truth signal it can act on. Where no such verifier exists, the agent cannot tell a plausible output from a correct one, and errors pile up across steps.
5. Least-privilege authority and staged autonomy
Credentials should be scoped to each task instance rather than to a service account. A policy layer (for example an OPA or Cedar decision point) should check each proposed action against permissions, value thresholds, and whether it can be reversed. A workable split:
| Action class | Control |
|---|---|
| Read, non-sensitive | Autonomous |
| Read, sensitive or personal data | Autonomous with logging and purpose binding |
| Write, reversible | Autonomous with compensating transaction defined |
| Write, irreversible or externally visible | Human confirmation before commit |
| Actions on OT or safety-instrumented systems | No direct write path; proposals only, via existing change control |
For infrastructure operators, the IT/OT boundary should be treated as absolute for autonomous write access, consistent with IEC 62443 zone and conduit modelling. An agent may prepare a change package; it does not execute it.
6. Adversarial assumptions about tool output
Anything a tool returns, a document yields, or a page scrape brings back is untrusted input that can carry instructions. Wherever an agent processes third-party content, expect indirect prompt injection as the normal condition. It is more than a residual risk. Mitigations that hold up: keep instructions structurally separate from data in the prompt; refuse to treat retrieved text as authorisation; allowlist egress so exfiltration has no channel; and never give one agent privileged data access, untrusted content ingestion and outbound communication all together. Where connectors are used, the tool definitions are themselves part of the trust boundary. They need version pinning and review, because definitions can change after approval.
7. Observability at trajectory granularity
Logging only the final output is not enough for diagnosis or assurance. The record must cover each step: prompt hash, model and version, tool invoked, arguments, result, latency, token count, policy decisions, and any human intervention. OpenTelemetry semantic conventions for generative AI give a workable schema. This record serves three separate purposes. It lets you rebuild what happened in an incident, and it feeds regression evaluation. It is also the evidence for audit under ISO/IEC 42001, or under the Article 12 logging obligations of the EU AI Act where the deployment falls in a high-risk class.
8. Evaluation against trajectories
Scoring only the output hides agents that reach acceptable answers by unsafe or expensive paths. A production evaluation harness should start with a fixed regression set of representative tasks with known-correct end states. It should score each step for tool selection accuracy and unnecessary actions. It should also track cost and latency distributions, include adversarial cases with injected instructions, and measure variance across repeated runs of identical inputs. Because agents are non-deterministic, comparing prompt or model versions on a single run tells you little. Pass rates across repeated trials are the figure that matters.
9. Failure and recovery design
Checkpointing state after each externally visible action lets the system resume rather than restart. Before autonomous operation, you need three things. Detect loops of repeated identical tool calls. Enforce step and spend ceilings outside the model. Define a handover format for escalating to a human operator. Where the operational context permits, the system should drop to a lower-autonomy mode.
How an agent run stops and hands over
Schematic of Foundation 9. Step counts and the ceiling of 12 are illustrative. All five controls sit outside the model; none depend on the agent noticing its own failure.
10. Deterministic code wherever determinism is available
Keep the agentic loop for steps that need judgement under ambiguity. Routing, validation, calculation, formatting, and sequencing of known workflows work better as conventional software that the agent calls. Systems that use model inference for deterministic sub-tasks pay in cost, latency, and error rates with no matching benefit. They also complicate the assurance argument, because the deterministic part can no longer be verified by inspection.
Assessment
The failure mode that keeps recurring in agentic deployments is capability-led design: a system built around what the model can do, with controls added afterward. The alternative is to specify the verification mechanism, the authority boundary, and the audit record first. Only then decide how much autonomy those limits will support. For government and infrastructure uses in particular, the second approach is also the only one that produces a defensible safety case.
Standards and references: IEC 62443, zone and conduit model for OT/IT segmentation; OpenTelemetry Semantic Conventions for Generative AI; ISO/IEC 42001, AI management systems; EU AI Act, Article 12 logging obligations for high-risk systems.
Free tool
TRACE Agent Evaluation
Score a candidate task against five criteria before you commit to an architecture. It is the practical test behind Foundation 1’s bounded task specification.
Free tool
Expert AI Agent Builder
Configure guardrails on an agent team, then red-team it against prompt injection, scope creep, and memory poisoning. These are the failure modes behind Foundations 5 and 6.
The AI Governance & ROI Executive Programme works through this discipline with engineering and risk leaders in the same room. It covers verification design, authority boundaries, and the audit record, before autonomy scope is ever expanded. Details are on the workshops page.
Cite or quote this article
Was this useful?
