Executive Summary
Agentic systems differ from single-shot inference in one crucial way: the model selects actions, observes consequences, and iterates. That changes where reliability actually comes from. It has less to do with how capable the underlying model is, and much more to do with the engineering built around it. Here are the ten elements that determine whether an agentic deployment holds up under real production conditions, particularly in regulated or safety-relevant environments.
Core conclusions
- A task only belongs to an autonomous agent if success can be checked programmatically and the agent runs against a hard step and cost budget. Everything else is decision support, not automation.
- The controls that matter most sit outside the model itself: narrow, validated tool schemas, least-privilege authority scoped per task, an absolute boundary at OT and safety-instrumented systems, and treating every tool result as untrusted input by default.
- Assurance requires a record, not a demo. That means trajectory-level logging of every step, evaluation against known-correct trajectories rather than final answers alone, and deterministic code wherever determinism is available.
The ten foundations, in ten slides
Save it, share it, or hand it to whoever is about to give an agent write access.
I have written before about the engineering checklist for shipping agentic systems to production once the architecture diagram is settled. This one sits a level below that: the foundational properties a system needs before “production” is even a reasonable word to use for it.
Foundations of dependable agentic AI
1. Bounded task specification
An agent requires a machine-checkable definition of success and an explicit termination condition. Where the objective cannot be verified programmatically, the task is a poor candidate for autonomous execution and should be restructured as decision support instead. The practical form is a task contract: input schema, permitted side effects, acceptance criteria, maximum step budget, and escalation trigger, stated up front. Absence of a termination condition is the single most common cause of runaway loops and unbounded cost.
2. Context as an engineered resource
Model performance degrades non-linearly as context fills with low-signal material. Effective designs treat the context window as a managed budget: retrieval that returns the minimum sufficient evidence, externalised state in a durable store rather than accumulated conversation history, periodic compaction of prior steps into structured summaries, and explicit provenance tags on retrieved content so the agent can distinguish instruction from data. Sub-agent delegation is principally a context isolation mechanism, not an organisational metaphor.
3. Tool surfaces designed for model consumption
Tool definitions are the primary control surface. Characteristics that measurably reduce failure rates:
- Narrow parameter schemas with enumerated values rather than free text, validated before execution.
- Idempotency keys on all write operations, so retries after timeout do not duplicate effects.
- Error returns that state the corrective action rather than a stack trace or an opaque status code.
- Token-efficient responses, with pagination and field selection, so a single call cannot exhaust the context.
- Consolidation of related operations into fewer, higher-level tools. Exposing a raw API surface of eighty endpoints reliably produces worse selection accuracy than fifteen task-shaped tools.
4. Closed-loop verification
The distinguishing property of agentic systems that improve with iteration is access to a deterministic verifier. Compilers, unit tests, schema validators, physics or hydraulic simulators, reconciliation checks against a system of record, and constraint solvers all provide a ground-truth signal the model can act on. Where no such verifier exists, the agent has no means of distinguishing a plausible output from a correct one, and error accumulates across steps.
5. Least-privilege authority and staged autonomy
Credentials should be scoped per task instance, not per service account, with a policy layer (for example an OPA or Cedar decision point) evaluating each proposed action against permissions, value thresholds, and reversibility. A workable partition:
| Action class | Control |
|---|---|
| Read, non-sensitive | Autonomous |
| Read, sensitive or personal data | Autonomous with logging and purpose binding |
| Write, reversible | Autonomous with compensating transaction defined |
| Write, irreversible or externally visible | Human confirmation before commit |
| Actions on OT or safety-instrumented systems | No direct write path; proposals only, via existing change control |
For infrastructure operators, the IT/OT boundary should be treated as absolute for autonomous write access, consistent with IEC 62443 zone and conduit modelling. An agent may prepare a change package; it does not execute it.
6. Adversarial assumptions about tool output
All content returned by a tool, retrieved from a document, or scraped from a page is untrusted input capable of carrying instructions. Indirect prompt injection is not a residual risk but the expected condition wherever an agent processes third-party content. Mitigations that hold up: structural separation of instructions from data in the prompt, refusal to treat retrieved text as authorisation, egress allowlisting so exfiltration has no channel, and avoidance of the combination of privileged data access, untrusted content ingestion, and outbound communication within a single agent’s authority. Where connectors are used, the tool definitions themselves are part of the trust boundary and require version pinning and review, as definitions can change after approval.
7. Observability at trajectory granularity
Logging the final output is insufficient for diagnosis or assurance. The required record covers each step: prompt hash, model and version, tool invoked, arguments, result, latency, token count, policy decisions, and any human intervention. OpenTelemetry semantic conventions for generative AI provide a workable schema. This record serves three separate purposes: incident reconstruction, regression evaluation, and the evidentiary basis for audit under ISO/IEC 42001 or Article 12 logging obligations of the EU AI Act where the deployment falls within a high-risk classification.
8. Evaluation against trajectories, not just outputs
Output-only scoring conceals agents that reach acceptable answers through unsafe or expensive paths. A production evaluation harness should include a fixed regression set of representative tasks with known-correct end states, step-level scoring for tool selection accuracy and unnecessary actions, cost and latency distributions rather than means, adversarial cases containing injected instructions, and measurement of variance across repeated runs of identical inputs. Non-determinism means single-run comparisons between prompt or model versions carry little information; pass rates across repeated trials are the meaningful figure.
9. Failure and recovery design
Checkpointing of state after each externally visible action permits resumption rather than restart. Loop detection on repeated identical tool calls, step and spend ceilings enforced outside the model, and a defined handover format for escalation to a human operator are all required before autonomous operation. The system should degrade to a lower-autonomy mode rather than halt, where the operational context permits.
10. Deterministic code wherever determinism is available
The agentic loop should be reserved for steps requiring judgement under ambiguity. Routing, validation, calculation, formatting, and sequencing of known workflows are better implemented as conventional software invoked by the agent. Systems that use model inference for deterministic sub-tasks incur cost, latency, and error rates without corresponding benefit, and they complicate the assurance argument, because the deterministic portion can no longer be verified by inspection.
Assessment
The recurring failure mode in agentic deployments is capability-led design: a system built around what the model can do, with controls retrofitted afterward. The alternative is to specify the verification mechanism, the authority boundary, and the audit record first, and only then determine what degree of autonomy those constraints will support. For government and infrastructure applications in particular, the second approach is also the only one that produces a defensible safety case.
Standards and references: IEC 62443, zone and conduit model for OT/IT segmentation; OpenTelemetry Semantic Conventions for Generative AI; ISO/IEC 42001, AI management systems; EU AI Act, Article 12 logging obligations for high-risk systems.
Free tool
TRACE Agent Evaluation
Score a candidate task against five criteria before you commit to an architecture, the practical test behind Foundation 1’s bounded task specification.
Free tool
Expert AI Agent Builder
Configure guardrails on an agent team, then red-team it against prompt injection, scope creep, and memory poisoning, the failure modes behind Foundations 5 and 6.
The AI Governance & ROI Executive Programme works through this discipline with engineering and risk leadership in the same room: verification design, authority boundaries, and the audit record, before autonomy scope is ever expanded. Details are on the workshops page.