Listen to this article
Executive Summary
Two agents can pass the same demo and be nowhere near equally trustworthy. One was assessed against a real job description, checked against the business outcome it’s meant to serve, put on a re-benchmarking schedule, and given a human reviewer who knows what a wrong answer looks like. The other was shipped because it worked on Tuesday. Building a trustworthy agent is closer to running a learning-and-development function than shipping software: assess proficiency before deployment, verify it against intent rather than a leaderboard, keep training it on a schedule, and always keep a way to pull it back to a known-good state.
40%+
of agentic AI projects will be canceled by the end of 2027, mostly from unclear ROI and inadequate risk controls, not model quality, Gartner (2025)
61% → 25%
GPT-4o’s success rate on identical retail-agent tasks, run once versus run eight times, τ-bench (Yao et al., 2024)
~7 months
the doubling time for the length of task a frontier agent can complete at 50% reliability, METR (2025), meaning last year’s proficiency test is already out of date
Dec 2027
when EU AI Act Article 14’s human-oversight obligations become enforceable for high-risk AI systems under Annex III
Core conclusions
- Passing a demo once is not proficiency. Reliability across repeated, identical attempts is the real bar, and most agents have never been tested against it.
- A capability benchmark ages out within months. A proficiency check done at go-live tells you almost nothing about the agent a year later.
- Human oversight and a rollback path are not compliance overhead. They are what makes it safe to keep letting the agent learn at all.
The five disciplines that make an agent trustworthy, ten slides
Save it, share it, or send it to whoever just approved an agent deployment on a single clean demo.
Agentic AI project failures usually trace back to trust
Gartner expects more than 40% of agentic AI projects to be canceled before the end of 2027. The stated reasons are escalating costs, unclear business value, and inadequate risk controls, not that the underlying models were incapable of the task. Gartner’s own analysts point to widespread “agent washing,” existing chatbots and robotic process automation relabelled as agents, and estimate that only a small fraction of the vendors claiming agentic capability actually have it.
That distinction matters for how you respond. If the problem were capability, the fix would be a better model. If the problem is trust, an unstructured deployment that nobody assessed, verified, or can pull back, then a better model changes nothing. The organisations still running agents in 2028 will not be the ones with the newest model. They will be the ones who built a process for deciding which agents earned production access and which didn’t.
An agent deserves the same onboarding discipline as a new hire
A new employee doesn’t get handed the keys on day one because they interviewed well. They get a job description, a probation period, and a manager who checks their work against real outcomes before responsibility expands. Agentic AI deserves the same discipline, and most deployments skip straight past it.
Start with the job description: the specific task, its boundaries, and what “done correctly” means in terms the business actually cares about, not just whether the output parses. Then assess proficiency before the agent touches anything live. This is where most evaluation still falls short. τ-bench, a 2024 benchmark for tool-using agents, found GPT-4o solving 61% of retail-agent tasks on a single attempt, but only 25% when the identical task was attempted eight times. The model wasn’t unreliable because it lacked knowledge. It was unreliable because a single pass says nothing about consistency, and consistency is what a business actually depends on. An agent that gets it right most of the time in testing but drifts under repetition hasn’t demonstrated proficiency, it’s demonstrated luck.
Free tool
TRACE Agent Evaluation
Score a candidate task against five criteria before it goes anywhere near production, the same pre-flight check a job description and probation review would give a new hire.
The second discipline most deployments skip is checking the agent against intent, not just accuracy. A customer-service agent that resolves 95% of tickets correctly but silently pushes every ambiguous case toward the answer that closes the ticket fastest has learned the wrong KPI. It looks proficient on a dashboard and is actively working against the outcome the business wanted. This is exactly the review a learning-and-development function runs on a new employee: not just “did they get the task done,” but “did they get it done the way we actually meant.”
Basic proficiency is only the starting point
Once an agent clears its initial assessment, the temptation is to consider the job finished. It isn’t, for a reason specific to this technology: the ground it was tested on moves. METR’s research on frontier agents found that the length of task an agent can complete at 50% reliability has been doubling roughly every seven months since 2019, driven less by raw model upgrades than by better reliability and tool use under the hood. A proficiency test run at deployment is measuring a capability profile that will look different within two quarters, sometimes because the underlying model provider pushed a silent update, sometimes because the agent’s own tools, data sources, or the tasks routed to it changed underneath it.
Continuous learning without continuous verification isn’t improvement, it’s drift you haven’t measured yet. The fix is the same one any L&D function already runs: put the agent back on the bench on a schedule, not just at launch. Re-run the original proficiency tests plus a set of harder, previously-unseen cases quarterly at minimum, and immediately after any change to the underlying model, its tools, or its data sources. Track the trend, not just the pass/fail, because a slow decline in success rate over several review cycles is a much more common failure mode than a sudden collapse, and it’s invisible if you only ever check whether the agent still clears the bar it cleared on day one.
Human-in-the-loop only works if the human knows what to look for
“Put a human in the loop” has become the default answer to agent risk, and it’s frequently implemented as someone glancing at outputs with no training on what a wrong one looks like, no defined authority to override, and no escalation path when they’re unsure. That version of human oversight breaks down precisely when it matters most, because agents that plan and call tools at machine speed can produce a plausible-looking wrong answer faster than an untrained reviewer can catch it.
Article 14 of the EU AI Act sets a higher bar than “someone is watching,” and it’s a reasonable design target even where it isn’t yet legally binding on you. It requires that human overseers understand the system’s actual capabilities and limitations, can recognise their own automation bias, can correctly interpret what the agent produced, and can override or halt it, not just flag it after the fact. Those obligations become enforceable for high-risk systems under Annex III from December 2027, and under Annex I from August 2028, which gives most organisations a runway to build the reviewer training now rather than backfilling it under deadline. Design the review role the way you’d design a manager’s role for a new hire: specific decision rights, specific triggers for escalation, and periodic calibration so two reviewers looking at the same output reach the same call.
Free tool
Board AI Oversight Checklist
Who actually has override authority over an agent’s decisions, what triggers escalation, and whether that person has ever been tested on it. Human-in-the-loop starts here, not with a dashboard.
Every agent deployment needs a rollback plan from day one
An employee who starts making bad calls gets pulled off the task, not deleted from the company, and the same principle should govern an agent that starts producing bad output because a data source went stale, a tool started returning malformed results, or a model update silently changed its behaviour. That requires three things most agent deployments don’t have on day one: a pinned, known-good version to roll back to, a monitoring signal that catches degradation before it becomes an incident, and a tested procedure for actually executing the rollback rather than discovering during an outage that nobody has done it before.
Version pinning matters more for agents than for typical software because the model underneath is often a vendor’s moving target, not a build you control, so “what changed” isn’t always visible from your side. Monitor for gradual drift, not just hard failures. A simple error-rate threshold catches an agent that broke outright, but the more common failure pattern is a slow decline in output quality that a fixed threshold misses entirely, which is why the same statistical-process-control thinking used in manufacturing quality control, tracking a metric against a dynamically calculated control limit rather than a fixed line, is now standard advice for production model monitoring. And whatever the mechanism, rehearse it. A rollback path that has never been executed is a hypothesis, not a plan.
Free tool
AI Model Performance & Health Dashboard
Track drift, latency and incident rate against thresholds designed to catch a slow decline, not just a hard failure, and see what a working rollback trigger actually looks like.
A practical starting checklist
None of this requires a research team. It requires treating agent deployment as an ongoing management discipline rather than a one-time technical rollout.
| Discipline | What good looks like | What to avoid |
|---|---|---|
| Skills assessment | Test against repeated, identical attempts (pass^k), not a single demo run | Approving on one clean pass-through |
| Intent verification | Check the agent’s outcome against the business goal it serves | Optimising for the metric that’s easiest to measure |
| Re-benchmarking | Scheduled, plus triggered by any model, tool, or data change | Testing once at go-live and never again |
| Human oversight | Trained reviewers with defined override authority and escalation triggers | An unstructured human “checking in” on outputs |
| Rollback readiness | A pinned known-good version and a rehearsed rollback procedure | Discovering the rollback plan doesn’t work during an incident |
This is an ongoing management practice with no finish line
None of the individual pieces here are exotic. Skills assessment, KPI verification, scheduled retraining, supervised review, and a tested fallback are exactly what a functioning L&D and performance-management process already does for a human employee. The only genuinely new problem is the pace: an agent’s competence can shift within a single quarter in ways a human’s rarely does, driven by a vendor update you didn’t ask for as often as by anything you changed yourself.
That’s the case for building this as a standing practice rather than a project with an end date. There is no proficiency level an agent reaches where the review process gets switched off, because the ground it’s being tested against keeps moving. The organisations that get real value out of agentic AI over the next few years won’t be the ones that deployed first. They’ll be the ones that never stopped treating deployment as day one of an ongoing job, not the finish line of a build.
Sources
- Gartner. (2025, June 25). Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027.
- Yao et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045.
- METR. (2025). Measuring AI Ability to Complete Long Software Tasks. arXiv:2503.14499.
- European Union. Article 14: Human Oversight, EU Artificial Intelligence Act.
- National Institute of Standards and Technology. (2023). AI Risk Management Framework (AI RMF 1.0).
The AI Governance & ROI Executive Programme builds this assessment-to-rollback structure around your own agents before they touch production, not after. Details are on the workshops page.
Was this useful?
