More than ten years in financial services, taking new products to clients through product, compliance, risk and technology teams.
BringsThe finance view: what a CFO can sign, book and review, from the baseline to the monthly cost bridge.
The series does not discuss Moomoo, and the views in it are not Moomoo's.
AI Benefit Realisation · Parts 1 to 4
This session
Five parts to this session.
The joint frameworkWhy AI benefits go unproven, and four controls the two offices sign together.Part 1
The baselineWhat to measure, for how long, and who signs before the first model call.Part 2
AttributionSeparating the AI's effect from everything else that changed.Part 3
The cost per outcomeTagging every model call, and the monthly AI cost review.Part 4
What to do this quarterActions, committee questions and what the series covers next.
AI Benefit Realisation · Parts 1 to 4
The argument in one slide
Most AI programmes cannot say what they returned.
25%
of AI initiatives delivered the return chief executives expected, and 16% scaled enterprise-wide
30%
of generative AI projects Gartner expected to be abandoned after proof of concept by the end of 2025
98%
of FinOps practitioners now manage AI spend, up from 31% in 2024
5-30×
more tokens per task for agentic models than for a standard chatbot
Sources: IBM CEO Study, May 2025; Gartner, July 2024 and March 2026 (forecasts); State of FinOps 2026.
AI Benefit Realisation · Parts 1 to 4
01
The joint framework.
The CAIO runs the work and the CFO owns the numbers. Neither owns the chain from starting point to realised benefit.
Part 1 · 10 October 2026
AI Benefit Realisation · Parts 1 to 4
Why AI benefits go unproven
The evidence of underperformance is consistent.
25%
of AI initiatives delivered expected ROI over the last few yearsIBM CEO Study, 2025, 2,000 CEOs
30%+
of generative AI projects abandoned after proof of concept by end-2025Gartner forecast, July 2024
40%+
of agentic AI projects cancelled by the end of 2027Gartner forecast, June 2025
63%
of finance organisations found AI implementation slower than expected in 2025Gartner, May 2026
"Counts of pilots, tools rolled out or use cases in production show that finance is moving, but they do not prove that AI is delivering the value." Gartner finance research, May 2026.
Gartner's reasons in 2024: poor data quality, inadequate risk controls, escalating costs or unclear business value. Grades: reported (surveys) and forecast.
AI Benefit Realisation · Parts 1 to 4
Belief and measurement
The best-known figures record what executives believe.
MIT NANDA · July 2025
"95% of organizations are getting zero return"
Based on 52 interviews, a survey of 153 leaders and a review of public announcements.
The authors' own caveat"Directionally accurate based on individual interviews rather than official company reporting."
Wharton and GBK · October 2025
Three in four leaders report positive returns
An executive survey, later the same year.
What both have in commonNeither is a measurement. The gap between belief and measurement is the problem this series addresses.
Grades: MIT NANDA reported, contested; Wharton reported, executive survey.
AI Benefit Realisation · Parts 1 to 4
The ownership gap
A benefit claim travels between two offices.
Chief AI Officer
Owns the use case, model, data and delivery team76% of organisations now have one, up from 26% a year earlier
Chief Financial Officer
Owns the budget, the benefits register and the board numbers62% of CFOs say their role expanded into technology or AI strategy
"The model cut handling time by a fifth."
The CFO cannot find the saving in the cost base.
Time was releasedand nobody decided what to do with it
The baseline was takenin a quiet month
The inference bill grewfaster than the saving
Sources: IBM CEO Study, May 2026; IBM CFO Study, September 2026 (reported, executive surveys). The three gaps are our reading of deployments we have seen.
AI Benefit Realisation · Parts 1 to 4
Four controls, signed jointly
One office prepares each control, and the other signs it.
ControlCAIO preparesCFO signsWhen
1Baseline
Metric definitions, data source, measurement period, natural variance
The baseline record and the target, as a finance figure
Before the first production model call
2Attribution
The method, the control group or comparison, the sample size
The grade of evidence each claimed benefit may carry
At pilot approval, before results exist
3Unit cost
Tagged inference, retrieval and evaluation spend per use case
The fully loaded cost per outcome, including human review
Monthly, from the first day of the pilot
4Stage-gates
The results against target, with method and confidence
The decision to scale, extend once, or stop
At each gate, on a fixed date
Our proposed split of responsibilities. The point is that neither office can pass a gate alone.
AI Benefit Realisation · Parts 1 to 4
Stage-gates that stop funding
Four gates, each with criteria written before the work.
Gate 0
Fund the pilot
Signed baseline, attribution method, target and kill threshold, cost ceiling, tagging in place.
Start, or not yet
Gate 1
Pilot review
Measured effect against target with method and confidence, quality guardrail, unit cost at pilot volume. Usually 8 to 12 weeks in.
Scale, extend once, or stop
Gate 2
Scale
Unit cost at full volume with token growth assumed; benefit per outcome above it by the agreed margin; entered in the AI inventory.
Scale or stop
Gate 3
Benefit realisation review
Realised against forecast at 6 and 12 months; the baseline still describes the business.
Keep or retire
A second extension at Gate 1 should need the investment committee.
Our proposal, adapted from Cooper's stage-gate systems (1990) and the UK government's Gateway reviews, where Gate 5 is dedicated to benefit realisation.
AI Benefit Realisation · Parts 1 to 4
Kill criteria
Four kill criteria for every pilot charter.
The measured effect, at the agreed confidence, is below the kill threshold.
The unit cost at projected volume exceeds the unit benefit.
The quality guardrail is breached and cannot be fixed within the extension.
The agreed attribution method could not be run, so the benefit cannot be measured.
Teams resist the fourth most, because the pilot may well be working. A benefit that cannot be measured cannot be defended to a board, an auditor or a regulator.
AI Benefit Realisation · Parts 1 to 4
Making stopping normal
A pilot stopped with a clean measurement has done its job.
Report the stop rate
A measure of the gates' health
A portfolio in which nothing has ever been stopped does not mean every pilot succeeded. It means the gates are not being used.
Gartner, 2026
"Cutting weak ideas early"
Gartner advised CFOs to stop looking for a single ROI formula and build a balanced portfolio. Mature finance uses such as payables automation generally return their investment within nine to ten months.
Sources: Gartner press releases, March and September 2026 (reported, analyst survey). The stop-rate measure is our proposal.
AI Benefit Realisation · Parts 1 to 4
The Singapore context
Singapore funds AI adoption and asks for outcomes.
Feb 2024
Budget 2024
More than S$1 billion over five years for AI compute, talent and industry.
Feb 2026
Budget 2026
National AI Missions, each "anchored in clear objectives and tangible outcomes".
YA 2027-28
Enterprise Innovation Scheme
AI spending qualifies, capped at S$50,000 a year. Untagged spend is harder to claim.
7 Oct 2026
MAS guidelines final
AI inventories "at an appropriate level of granularity", and board oversight.
7 Oct 2027
MAS takes effect
Some sections follow a year later.
Today
No benefits guidance
We found none for AI in Singapore. The UK's Gate 5 review is the closest model.
A joint CFO and CAIO gate, with written criteria and a record of every decision, is evidence of the oversight MAS expects.
Sources: MDDI, February 2024; Budget 2026 statement; MAS media release, October 2026. Hollow points are dates still to come, or our finding.
AI Benefit Realisation · Parts 1 to 4
02
The baseline.
A baseline taken after the pilot starts is a reconstruction, and reconstructions flatter the pilot.
Part 2 · 17 October 2026
AI Benefit Realisation · Parts 1 to 4
An old habit, rarely applied to AI
Few organisations measure AI against a starting point.
<1 in 5
organisations tracked KPIs for their gen AI solutions. KPI tracking had the most bottom-line impact of the twelve practices tested.McKinsey State of AI, March 2025
37%
of organisations attribute any EBIT impact to AI, essentially unchanged from the year beforeMcKinsey State of AI, August 2026
Initiation
"should be taken during project initiation so that performance is not impacted by any element of project implementation"UK IPA benefits guide, on the baseline
Grades: McKinsey reported, executive surveys; UK Infrastructure and Projects Authority guide, 2017, government guidance.
AI Benefit Realisation · Parts 1 to 4
Why a late baseline flatters the pilot
Four reasons, and none needs bad faith.
01
People misremember the starting point
Capable people who had just done the work misjudged the AI's effect by a wide margin.
02
Pilots go where performance is worst
Regression to the mean: an unusually bad month is followed by a better one, with or without AI.
03
New tools look better at first
Novelty effects: early results differ from the settled effect because users react to the change itself.
04
Measures shift once the project is known
Teams tidy data, change a process or redefine a metric "in anticipation of the intervention".
Sources: METR, 2025; Barnett et al., 2005; Kohavi, Tang and Xu, 2020; HM Treasury guidance on evaluating AI interventions, 2024. The grouping is ours.
AI Benefit Realisation · Parts 1 to 4
METR, July 2025 · randomised, 16 developers
They were slower, and believed they were faster.
0
Expected before the study
24% faster
Believed afterwards
20% faster
Measured
19% slower
METR says the results no longer reflect current tools. The perception gap is the point: a baseline gathered from recollection inherits the same error.
Source: METR, July 2025, and its February 2026 note. Grade: measured, randomised experiment, early-2025 tools.
AI Benefit Realisation · Parts 1 to 4
The same eight pilot weeks, two baselines · illustrative
The reconstructed baseline nearly triples the gain.
Against the signed baseline
3%
better. Twelve weeks including a quarter-end. The pilot goes back for a longer run.
Against a reconstruction
9%
better. Three busy weeks everyone remembers. The pilot gets scaled.
Average handling time, minutes. Illustrative example built for the post: the signed baseline covers twelve weeks including a quarter-end; the reconstruction uses the three busy weeks everyone remembers.
AI Benefit Realisation · Parts 1 to 4
Choosing what to measure
Pair every speed measure with a quality measure.
Use case
Primary outcome
Quality guardrail
System of record
Contact-centre assistant
Calls resolved per agent hour
Repeat contact within 7 days; complaint rate
Telephony and case-management timestamps
Invoice processing
Cost per invoice processed
Payment errors; duplicate payments caught late
Accounts payable ledger and workflow logs
Claims assessment
Claims closed per assessor per week
Reopened claims; leakage found at audit
Claims system and audit sample
Credit memo drafting
Days from application to credit decision
Memos returned by the credit committee
Loan origination system
Month-end close
Working days to close
Post-close adjustments
General ledger and close calendar
Our examples. In 5,172 support agents, productivity rose 15% on average and 30% for novices, while the most experienced saw "small gains in speed and small declines in quality" (Brynjolfsson, Li and Raymond, QJE 2025).
AI Benefit Realisation · Parts 1 to 4
Natural variance decides the pilot length
A 5% target can hide inside the noise.
Smallest detectable change ≈ 2.8 × weekly spread × √(1/baseline weeks + 1/pilot weeks)80% power at 5% significance · weekly spread 6% of the average
Business case target 5%
8.4%
8 weeks + 8 weeks
5.9%
16 weeks + 16 weeks
4.2%
32 weeks + 32 weeks
The rule and the 8.4% example are from the post (standard two-sample power calculation, simplified). The 16- and 32-week bars are our arithmetic with the same rule. Correlated or trending weeks make the real figure larger.
AI Benefit Realisation · Parts 1 to 4
When the target sits inside the noise
Four responses, decided before approval.
01
Run longer
More weeks narrow the band, but slowly. Doubling both periods cuts the detectable change by only about 30%.
02
Add a control group
A staggered rollout or holdout measures the same weeks, so shared movement cancels out. Part 3.
03
Use the baseline in the analysis
CUPED cut variance "by about 50%" at Bing. It needs a baseline at team or case level, recorded before the pilot.
04
Accept it cannot be measured
A legitimate finding, and the fourth kill criterion. Cheaper at approval than at Gate 1.
Source for CUPED: Deng, Xu, Kohavi and Walker, WSDM 2013 (measured, company experiment platform).
AI Benefit Realisation · Parts 1 to 4
The one-page baseline record · contact-centre example
The baseline fits on one page, signed by finance.
Use case identifierCC-ASSIST-01, the same ID used to tag model calls and costs
Primary outcomeCalls resolved per agent hour, from the case system
Data source and ownerTelephony platform and CRM; Head of Contact Centre Operations
Measurement period13 weeks to 30 September, including quarter-end
Average and weekly spread4.1 calls per hour; weekly spread 6%
Trend and seasonalityFlat trend; quarter-end weeks 8 to 10% lower
Smallest detectable change7.5% before-and-after over an eight-week pilot
Quality guardrailRepeat contacts within 7 days no more than 12%
Expected disbenefitsEscalations to supervisors; supervisor review hours
Current cost to serveS$9.40 per resolved call
Target and kill thresholdTarget 12%; kill below 5%
Plan for released timeAbsorb volume growth for two quarters, then no backfill of leavers
Signatures and freeze dateSigned and frozen before the first production model call
Change logEmpty at signature; every later change logged with a reason
Our proposed template. The example figures are illustrative.
AI Benefit Realisation · Parts 1 to 4
Who signs and when
Six steps to the first production model call.
1 · CAIO drafts, finance agrees
Define the outcome
One primary measure, a quality guardrail and the cost to serve.
2 · Finance or data owner
Pull the history
A full business cycle, by someone outside the pilot team.
3 · CAIO team
Measure the spread
Average, spread, trend and peaks; the smallest detectable change.
4 · Both offices
Set target and kill threshold
Both above the detectable change.
5 · CFO signs
Sign and freeze
Later changes go in the change log with a reason.
6 · Gate 0 passed
First model call
Measurement continues on the same definitions.
MAS's final guidelines expect AI performance thresholds to be "clearly defined, documented, and mutually agreed upon by business owners, developers and reviewers". The signed baseline applies the same discipline to the benefit side.
Sequence: our proposal. MAS Guidelines on AI Risk Management, 7 October 2026, para 5.14(a). No Singapore guidance requires a benefits baseline for AI today.
AI Benefit Realisation · Parts 1 to 4
When there is no clean history
Three better answers than a reconstruction.
01
A shadow measurement period
Instrument the process and measure it for four to eight weeks before the AI goes live.
It delays the pilot. It is the only way to get a baseline on the same definition as the results.
02
Sampled measurement
Measure a random sample of cases by hand, such as the time taken on a hundred randomly chosen claims.
A small random sample is more reliable than a large estimate.
03
A control group from day one
If history cannot be measured, measure a comparison group during the pilot.
The concurrent group becomes the reference point.
Never accepted: the pilot team's estimates, vendor benchmarks or another organisation's published results.
AI Benefit Realisation · Parts 1 to 4
Time released needs a plan
Most pilots release time before they release cash.
~3%
of work hours saved by Danish workers using AI chatbots, self-reported. The study found "precise null effects on earnings and recorded hours", ruling out effects above 2%.
Humlum and Vestergaard, NBER WP 33777, revised March 2026
State the type of benefit in the baseline
Capacityto absorb growth
Cost avoidedhiring that no longer happens
Cost removedspend that leaves the budget
Revenueincome the AI adds
Time released is reported as capacity until cost leaves the budget.
Grade: measured, administrative data and survey. The four benefit types and the reporting rule are our proposal.
AI Benefit Realisation · Parts 1 to 4
03
Attribution.
Revenue rose and the AI went live in the same quarter. The only reliable answer is a comparison group measured over the same weeks.
Part 3 · 24 October 2026
AI Benefit Realisation · Parts 1 to 4
One pilot, read two ways · illustrative
A 12% gain becomes 5 points next to a comparison group.
Before and after
12%
faster in the pilot teams, before and after.
Same weeks, no AI
7%
faster anyway, because the routing change helped everyone.
The AI's effect is the difference: about 5 points.
Handling time indexed to 100 before go-live. Illustrative example built for the post: call routing was redesigned in the same quarter and helped every team.
AI Benefit Realisation · Parts 1 to 4
Five methods, strongest first
The methods give very different confidence.
Randomised holdout
Some teams or cases are randomly kept without the AI. The difference is the effect.
Highest
Staggered rollout
Waves; each compared with the teams still waiting.
High
Matched comparison
Similar teams without the AI, chosen before the pilot.
Moderate
Trend-adjusted before and after
Against the baseline, corrected for trend and season.
Low
Self-reported estimate
Useful for deciding what to measure next.
Not counted
Our ranking. Every claimed benefit carries its method and grade, and benefits of different grades are reported separately, never summed.
AI Benefit Realisation · Parts 1 to 4
Randomised holdout
Three design decisions go in the pilot charter.
01
The unit of randomisation
Where the AI is used and its effects stay contained: the individual for a drafting assistant, the team for a tool that changes how work is shared.
Randomising whole teams or sites prevents most leakage, such as held-out agents copying drafts from colleagues.
02
The size of the holdout
Large enough to detect the target. 10 to 20% of units is common in practice.
The right figure comes from the detectable-change calculation in Part 2.
03
The cost
The held-out group goes without the benefit for the pilot. That forgone benefit is the price of the evidence.
Usually small next to scaling a tool on a number that turns out to be wrong.
Leakage and interference: Kohavi, Tang and Xu, 2020. The holdout share is our practice.
AI Benefit Realisation · Parts 1 to 4
Cui et al., Management Science · three randomised trials
Even large, well-run trials give a wide range.
4,867software developers
3experiments, at Microsoft, Accenture and a Fortune 100 firm
Noisyeach experiment, on its own
0%12.5%25%37.5%50%
6%
46%
Roughly 6 to 46%, at 95% confidence
26%more completed tasks
A single-team pilot without a comparison group could not produce a reliable estimate at all.
Estimate 26.08%, standard error 10.3 points (measured, randomised experiments). The interval is our arithmetic from the published estimate and standard error.
AI Benefit Realisation · Parts 1 to 4
Dell'Acqua et al., Organization Science 2026 · 758 BCG consultants
Randomisation shows the harms a before-and-after hides.
Inside the AI frontier
12.2%
more tasks completed
25.1%
faster
With significantly better quality.
Outside the AI frontier
19%
less likely to reach a correct answer
Without the random assignment, this finding would have been invisible. A pilot that measured only the tasks the AI handled well would have reported a clean gain.
Grade: measured, randomised experiment.
AI Benefit Realisation · Parts 1 to 4
Dillon et al., 2025 · 7,137 workers in 66 firms, randomised
Book the effect per licence paid for.
80% used the tool: about two fewer hours a week on email20% did not use it, and the licence was still paid
2 hoursper user
×
80%who used it
=
1.6 hoursper licence, roughly
Statisticians call the per-licence figure the intention-to-treat effect. It is the one a business case needs, because the organisation pays per licence. The vendor's case study will usually quote the larger number.
Source: Dillon, Jaffe, Immorlica and Stanton, arXiv 2504.11436 v4 (measured, randomised field experiment). 1.6 hours is our arithmetic, assuming non-users save nothing.
AI Benefit Realisation · Parts 1 to 4
A common shortcut
Comparing heavy and light users does not measure the tool.
What the comparison mixes
Heavy users chose to use the tool, and they are often the people who were already faster, more curious or less busy. The difference mixes the AI's effect with the difference between the people.
What it is good for
Deciding where to train. It should never be booked as a benefit.
AI Benefit Realisation · Parts 1 to 4
Staggered rollout · schematic
Each wave is compared with the teams still waiting.
Month 1Month 2Month 3Month 4Month 5Month 6
Wave 1
Wave 2
Wave 3
Wave 4
Live with the AIStill waiting: the comparison for that month
Draw the wave order by lotIt becomes a stepped-wedge trial, with nearly the confidence of a holdout.
Compare only with teams not yet startedEarly teams used as the comparison bias the standard regression when effects grow over time (Goodman-Bacon, 2021).
Brynjolfsson, Li and Raymond used a staggered rollout for 5,172 agents: 15% average gain, 30% for novices. Estimators: Callaway and Sant'Anna, 2021. Wave order rule: HM Treasury, Magenta Book.
AI Benefit Realisation · Parts 1 to 4
When the AI cannot be split by user
Randomise by time block, or build a synthetic comparison.
Switchback test
For routing, pricing and dispatch
Run the AI in some time blocks and the old rules in others, with the schedule drawn at random, across several sites.
The catchThe number of independent blocks decides the precision, so switchbacks need weeks of blocks. DoorDash described the method for its dispatch algorithm in 2018.
Synthetic control
For a change in one unit
A weighted mix of untreated units whose history matches the treated unit before the change, such as the other countries of a regional insurer.
The ruleFix the weights before the results are seen. A synthetic control chosen afterwards can be made to show almost anything.
Sources: Kastelman and Ramesh, DoorDash Engineering, 2018 (reported); Abadie, Diamond and Hainmueller, JASA 2010 (measured, methods literature).
AI Benefit Realisation · Parts 1 to 4
The weakest counted method, and overlapping changes
When nothing else is possible, say so in the grade.
Trend-adjusted before and after
Interrupted time series
Fit the weekly history, project it forward, and measure the departure. It needs a long regular history, an expected shape set in advance, and seasonal adjustment.
Where it failsThe reorganisation, the price change and the AI launch are announced together. A control series helps considerably.
Several changes at once
Do not split the total by negotiation
Every sponsor would then claim the full effect elsewhere.
Three better optionsSequence the changes; use a factorial design; or report one combined benefit with one owner.
Sources: Lopez Bernal, Cummins and Gasparrini, 2017; HM Treasury, Magenta Book Annex A, 2026.
AI Benefit Realisation · Parts 1 to 4
Choosing a method for each use case
The method follows from how the AI's effects spread.
Use case
Unit of comparison
Recommended method
Fallback
Drafting assistant for relationship managers
Individual or team
Randomised holdout
Staggered rollout, randomised wave order
Contact-centre agent assistant
Team or site
Staggered rollout, randomised wave order
Matched comparison of similar sites
Call routing or pricing model
Site and time block
Switchback test
Trend-adjusted, with a control series
Claims triage in one country of a regional insurer
Country
Synthetic control from the other countries
Trend-adjusted before and after
Organisation-wide finance close assistant
The whole organisation
Trend-adjusted before and after
Report as capacity, graded low
Our examples. The recommended method is the strongest the use case usually allows; the fallback applies when the first is impractical.
AI Benefit Realisation · Parts 1 to 4
What each method costs
Finance should see the method's cost at approval.
Method
Main cost
Grade
Randomised holdout
Benefit forgone by the holdout during the pilot
Highest
Staggered rollout, randomised order
Later waves wait; analyst time for a modern estimator
High
Switchback test
Weeks of blocks to reach enough independent units
High
Synthetic control
Data preparation; weights fixed in advance
Moderate
Matched comparison
Risk that the groups differ in ways nobody measured
Moderate
Trend-adjusted before and after
Confounding by anything launched at the same time
Low
A sponsor who asks to waive a holdout states the alternative and the grade it supports, in the charter. A waiver does not remove the evidence requirement.
AI Benefit Realisation · Parts 1 to 4
04
The cost per outcome.
The price per token falls every quarter while the AI bill rises. Account for it like any other variable cost.
Part 4 · 29 October 2026
AI Benefit Realisation · Parts 1 to 4
Why the bill rises while prices fall
Prices fall fast, and consumption grows faster.
9-900×
yearly fall in the price of a fixed level of performance, depending on the taskEpoch AI, March 2025
~40×
a year, to match GPT-4 on PhD-level science questionsEpoch AI, March 2025
5-30×
more tokens per task for agentic models than a standard chatbotGartner forecast, March 2026
>5×
rise in inference cost per agentic workflow through 2028, the "inference paradox"Gartner forecast, 2026
Nobody would accept "the price per kilowatt-hour fell" as an explanation of a rising electricity bill.
Grades: Epoch measured, benchmark price analysis; Gartner forecasts, read through syndicated copies of the release text.
AI Benefit Realisation · Parts 1 to 4
The bill is a product of three numbers
Only one of the three is falling.
▲
Volume
More people, more use cases, more calls as each becomes daily work.
×
▲
Tokens per task
Retrieval adds input, reasoning models bill their working, agents call the model many times per task.
×
▼
Price per token
Falls at fixed performance. Teams move to newer, pricier models, and a newer tokeniser can count about 30% more tokens for the same text.
=
▲
The AI bill
Rises, unless someone explains it each month.
Tokeniser: Anthropic pricing documentation, October 2026 (newer tokeniser, Claude 4.7 onwards). Agent token use: Gartner, March 2026 (forecast).
AI Benefit Realisation · Parts 1 to 4
One ID on both sides of the ledger
The benefit and the cost carry the same key.
Benefits register
Use caseCC-ASSIST-01
OwnerHead of Contact Centre Ops
Benefit per reply sentS$1.68
Method and gradeStaggered rollout · High
AI gateway cost log
Use caseCC-ASSIST-01
Calls this monthTagged at the call
Full running costS$19,000
Cost per reply sentS$0.48
Unit margin, a simple joinS$1.20 per reply
Figures from the post's illustrative worked example. Shared costs are allocated by a driver agreed before the quarter starts.
AI Benefit Realisation · Parts 1 to 4
Four tagging rules
Tagging is a small engineering job with four rules.
01
Tag at the call
One internal gateway rejects calls without a use-case ID and records tokens, model and cost. A key per use case gives a second check.
02
Allocate shared costs by a declared driver
Requests or tokens, chosen before the quarter, and not changed when a use case's numbers look poor.
03
Keep build and run apart
Development, testing and evaluation runs are build cost, reported apart from the live service.
04
Show unused commitments
The unused part of committed spend is its own platform line, so a purchasing decision does not hide in unit costs.
FinOps Foundation's FOCUS specification (1.3, December 2025) gives billing data from several providers one format, and the Foundation has a workstream on AI token charges.
AI Benefit Realisation · Parts 1 to 4
The cost-per-outcome numerator · illustrative
Model tokens are 32% of the running cost.
Model tokensIncluding failed calls and retries
6,100
Retrieval and computeVector database, search, orchestration
1,800
Monitoring and guardrailsLogging, filters, tracing, alerts
900
Evaluation runsScheduled quality tests
700
Oversight and upkeepReviewers sampling drafts; prompt and knowledge-base staff
7,400
Shared platformGateway and platform team, by requests
2,100
Total per month, S$19,000
Illustrative worked example: a drafting assistant for customer-email replies. Count reviewer time here only if it sits outside the process whose benefit was measured.
AI Benefit Realisation · Parts 1 to 4
The denominator · illustrative
Divide by the outcome the benefit was measured in.
S$19,000
57,000 drafts
S$0.33
Flatters the use case: a draft nobody sends still costs money.
S$19,000
40,000 replies sent
S$0.48
The unit the benefit was measured in.
Unit margin per reply
Benefit: 2.4 min × S$0.70S$1.68
Cost per reply sent−S$0.48
Earned per replyS$1.20
Illustrative. A use case whose cost per outcome is above its benefit per outcome loses money on every transaction, however impressive the total benefit looks.
AI Benefit Realisation · Parts 1 to 4
Explaining the change each month · illustrative
A bridge splits the change into volume, consumption and rate.
Replies rose from 32,000 to 40,000; retrieval was widened; prompt caching was turned on. Cost per reply rose from about S$0.44 to S$0.48.
AI Benefit Realisation · Parts 1 to 4
The share nobody has tagged, and the levers
Untagged spend should fall every month.
Where untagged spend comes from
Keys that predate the gateway
Personal accounts expensed to a cost centre
AI features bundled into subscriptions
Pilots that skipped the register
Aim: below a tenth within two quarters. Each line is tagged, registered with an owner, or switched off.
Levers that lower cost per outcome
Route by difficultySmall model for simple tasks. Gartner's main defence against the inference paradox.
Prompt cachingCache reads at 10% of base input price on most current Anthropic models.
Batch processing50% cheaper for work that can wait.
Limit outputOutput tokens cost five times input.
Cap agent stepsRemoves the long tail of runaway tasks.
Prices from Anthropic's pricing documentation, October 2026; other providers differ. US-only inference costs 1.1×: a governance choice whose cost belongs in the line. The one-tenth aim is our judgement.
AI Benefit Realisation · Parts 1 to 4
The monthly AI cost review
One page, three numbers, owners present.
01
Cost per outcome against benefit per outcome
One row per live use case, with last month and the trend.
A negative unit margin is flagged for the next stage-gate.
02
The bridge
Volume, consumption per outcome and rate.
One line from the owner for any consumption change over 10%.
03
The untagged share
Percentage of AI spend with no use-case ID.
With the three largest untagged items.
Two habits from the finance sideAccrue AI costs from gateway logs at month end; do not wait for the invoice.Budget and alert on cost per outcome, as well as total spend.
Our proposal. The 10% threshold and the accrual habit are our practice.
AI Benefit Realisation · Parts 1 to 4
05
What to do this quarter.
The controls are cheap. The expensive thing is scaling a use case on a number nobody can defend.
AI Benefit Realisation · Parts 1 to 4
Seven actions
Seven actions for this quarter.
1Name the joint ownersA delivery owner and a finance owner for every use case.
2Baseline before the next pilotThe one-page record, from systems of record, signed by finance.
3Agree attribution at approvalMethod, unit, estimator and grade in the pilot charter.
4Tag every model callThe register ID on every call, from day one.
5Build the cost-per-outcome lineInference, retrieval, evaluation, review and platform share.
6Write the gates and kill criteriaFixed dates; one extension without the committee.
7Report benefits by gradeHoldout and rollout results in one column, estimates in another, never summed.
AI Benefit Realisation · Parts 1 to 4
Questions for the investment committee
Five questions to put to management in writing.
Which of our AI use cases had a signed baseline before they started?
For each claimed benefit, which group over which weeks tells us what would have happened without the AI?
Is each figure per licence paid for, or per active user?
What does one outcome cost us today for each use case, and what share of our AI spend has no owner?
When did we last stop funding an AI pilot, and what did we learn from it?
Drawn from the question lists at the end of each part.
AI Benefit Realisation · Parts 1 to 4
What to take away
One lesson from each part.
01
Make the chain a joint product
Four controls, each prepared by one office and signed by the other, so neither can pass a gate alone.
02
Sign the baseline first
One page, a full business cycle, the natural spread, and a plan for released time, frozen before the first model call.
03
Choose the comparison first
A group measured over the same weeks, written into the charter before results exist, booked per licence.
04
Put the cost on the benefit's line
Same ID, same outcome unit, reviewed monthly on three numbers.
AI Benefit Realisation · Parts 1 to 4
Where we could be wrong
Four limits we set out in the series.
Part 1 · strategic bets
Unit economics can stop the wrong work
New products and market position need a gate judged on option value.
Part 2 · holdouts everywhere
The historical baseline would matter less
We would still keep the record for the cost to serve, the guardrail and released time.
Part 3 · cheap pilots
Precision may cost more than it is worth
A low-graded comparison may do, as long as the grade travels with the number.
Part 4 · small estates
A gateway and bridge may be overkill
We would still tag every call from the start; adding tags later means rebuilding the history.
AI Benefit Realisation · Parts 1 to 4
Evidence and methodology · 1 of 2
How strong each claim is: surveys and forecasts.
Claim
Source
Grade
25% of AI initiatives delivered expected ROI; 16% scaled
IBM CEO Study, May 2025, 2,000 CEOs
Reported, executive survey
At least 30% of GenAI projects abandoned after proof of concept by end-2025
Gartner, July 2024
Forecast
"95% of organizations are getting zero return"
MIT NANDA, July 2025; 52 interviews, 153 survey responses
Reported, contested
76% have a CAIO, up from 26%; 62% of CFOs' roles expanded into AI
IBM CEO Study, May 2026; IBM CFO Study, September 2026
Reported, executive surveys
Less than one in five tracked KPIs for gen AI; 37% attribute any EBIT to AI
McKinsey State of AI, March 2025 and August 2026
Reported, executive surveys
98% of practitioners manage AI spend, from 31% in 2024
State of FinOps 2026, 1,192 respondents
Reported, practitioner survey
Price at fixed performance falls 9x to 900x a year
Epoch AI, March 2025
Measured, with the authors' caveat
5 to 30x more tokens per task for agents; >5x cost per agentic workflow through 2028
Gartner, March 2026 and 2026
Forecast
MAS AI risk guidelines final 7 October 2026; thresholds mutually agreed
MAS Guidelines on AI Risk Management, para 5.14(a)
Measured, regulator record
No Singapore guidance requires an AI benefits baseline
Our search of MAS, IMDA and GovTech publications
My call
AI Benefit Realisation · Parts 1 to 4
Evidence and methodology · 2 of 2
Experiments, guidance and our own proposals.
Claim
Source
Grade
758 consultants; 12.2% more tasks, 25.1% faster; 19% less likely correct outside the frontier
Dell'Acqua et al., Organization Science, 2026
Measured, randomised experiment
5,172 agents; 15% average, 30% for novices; quality declines for most experienced
Brynjolfsson, Li and Raymond, QJE, 2025
Measured, staggered rollout
16 developers, 19% slower, believed 20% faster
METR, July 2025; METR says results are out of date
Measured, early-2025 tools
26.08% more tasks, standard error 10.3%, 4,867 developers
Cui et al., Management Science, 2026
Measured, randomised experiments
80% of treated workers used the tool; about two fewer hours a week on email
Dillon et al., arXiv 2504.11436 v4, 2025
Measured, randomised field experiment
CUPED cut variance by about 50% at Bing
Deng, Xu, Kohavi and Walker, WSDM 2013
Measured, company platform
About 3% of hours saved; no earnings or hours effects above 2%
Humlum and Vestergaard, NBER WP 33777, rev. March 2026
Measured, administrative data
Baseline at initiation; evaluation methods and parallel trends
UK IPA benefits guide, 2017; HM Treasury Magenta Book, 2026
Measured, government guidance
3% against 9%; 12% against 5 points; S$19,000 cost line and bridge
Where a study has a published version and an earlier working paper with different numbers, we used the published one. The worked examples are illustrative, and the decision rights, baseline record, gates, cost lines and review are our proposal, adapted from UK benefits and evaluation guidance.
AI Benefit Realisation · Parts 1 to 4
For discussion
Four questions for the room.
Which AI benefit in our last board pack would survive an audit committee's questions about its method?
If we froze a baseline for our next pilot today, who would pull the data, and who would sign it?
Which pilot would we stop tomorrow if we applied our own kill criteria honestly?
Could we say this month which use case ran which part of our AI bill?
AI Benefit Realisation · Parts 1 to 4
Sources · 1 of 3
Works cited in Part 1.
1IBM. (2025). IBM study: CEOs double down on AI while navigating enterprise hurdles.newsroom.ibm.com
2Gartner. (2024). Gartner predicts 30% of generative AI projects will be abandoned after proof of concept by end of 2025.gartner.com
3Gartner. (2025). Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027.gartner.com
4Challapally, A., Pease, C., Raskar, R., and Chari, P. (2025). The GenAI divide: state of AI in business 2025. MIT NANDA.webberwentzel.com
5Wharton School and GBK Collective. (2025). Accountable acceleration: gen AI fast-tracks into the enterprise.knowledge.wharton.upenn.edu
6Gartner. (2026). Finance AI press releases, 24 March, 28 May and 24 September 2026.gartner.com
7IBM. (2026). IBM study: CEOs are reshaping C-suite roles for the AI era.newsroom.ibm.com
8IBM. (2026). IBM study: as AI scales enterprise-wide, CFOs play an expanded role in transformation.newsroom.ibm.com
9Infrastructure and Projects Authority. (2017). Guide for effective benefits management in major projects.apm.org.uk
10Dell'Acqua, F., et al. (2026). Navigating the jagged technological frontier. Organization Science, 37(2), 403-423.doi.org
11Brynjolfsson, E., Li, D., and Raymond, L. (2025). Generative AI at work. Quarterly Journal of Economics, 140(2), 889-942.doi.org
12DBS Group. (2026). Annual report 2025: letter from the chairman and CEO.dbs.com
13Stanford HAI. (2025). AI Index report 2025.hai.stanford.edu
14Epoch AI. (2025). LLM inference prices have fallen rapidly but unequally across tasks.epoch.ai
15Gartner. (2026). Gartner predicts that by 2030 performing inference on an LLM with 1 trillion parameters will cost GenAI providers over 90% less than in 2025.gartner.com
16FinOps Foundation. (2026). State of FinOps 2026.data.finops.org
AI Benefit Realisation · Parts 1 to 4
Sources · 2 of 3
Works cited in Parts 1 and 2.
17FinOps Foundation. (2026). FinOps for AI overview.finops.org
18Storment, J.R. (2026). Token economics: the atomic unit of AI value. FinOps Foundation.finops.org
19FinOps Foundation. FOCUS: the FinOps open cost and usage specification.focus.finops.org
20Linux Foundation. (2026). Linux Foundation announces Tokenomicon, a new conference for the economics of AI.linuxfoundation.org
25Monetary Authority of Singapore. (2026). MAS sets out supervisory expectations on responsible AI adoption by financial institutions.mas.gov.sg
26Ministry of Digital Development and Information. (2024). AI initiatives launched to uplift Singapore's economic potential.mddi.gov.sg
27Ministry of Finance, Singapore. (2026). Budget 2026 statement: harness AI as a strategic advantage.singaporebudget.gov.sg
28HM Treasury. (2026). The Magenta Book: central government guidance on evaluation.gov.uk
29Frontier Economics for HM Treasury. (2024). Guidance on the impact evaluation of AI interventions.assets.publishing.service.gov.uk
30McKinsey & Company. (2025). The state of AI: how organizations are rewiring to capture value.mckinsey.com
31McKinsey & Company. (2026). The state of AI in 2026: on the road to ROI.mckinsey.com
32METR. (2025). Measuring the impact of early-2025 AI on experienced open-source developer productivity.metr.org
AI Benefit Realisation · Parts 1 to 4
Sources · 3 of 3
Works cited in Parts 2, 3 and 4.
33METR. (2026). We are changing our developer productivity experiment design.metr.org
34Barnett, A.G., van der Pols, J.C., and Dobson, A.J. (2005). Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology, 34(1), 215-220.doi.org
35Kohavi, R., Tang, D., and Xu, Y. (2020). Trustworthy online controlled experiments: a practical guide to A/B testing. Cambridge University Press.doi.org
36Deng, A., Xu, Y., Kohavi, R., and Walker, T. (2013). Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. WSDM 2013.doi.org
37Monetary Authority of Singapore. (2026). Guidelines on artificial intelligence risk management.mas.gov.sg
38Humlum, A., and Vestergaard, E. (2026). Still waters, rapid currents: early labor market transformation under generative AI. NBER Working Paper 33777.nber.org
39HM Treasury. (2026). Magenta Book Annex A: analytical methods for use within an evaluation.gov.uk
40Cui, K.Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., and Salz, T. (2026). The effects of generative AI on high-skilled work: evidence from three field experiments with software developers. Management Science.doi.org
41Dillon, E.W., Jaffe, S., Immorlica, N., and Stanton, C.T. (2025). Shifting work patterns with generative AI. arXiv 2504.11436.arxiv.org
42Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254-277.doi.org
43Callaway, B., and Sant'Anna, P.H.C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200-230.doi.org
44Kastelman, D., and Ramesh, R. (2018). Switchback tests and randomized experimentation under network effects at DoorDash. DoorDash Engineering.careersatdoordash.com
45Abadie, A., Diamond, A., and Hainmueller, J. (2010). Synthetic control methods for comparative case studies: estimating the effect of California's tobacco control program. Journal of the American Statistical Association, 105(490), 493-505.doi.org
46Lopez Bernal, J., Cummins, S., and Gasparrini, A. (2017). Interrupted time series regression for the evaluation of public health interventions: a tutorial. International Journal of Epidemiology, 46(1), 348-355.doi.org
47Gartner. (2026). Gartner predicts AI inference costs per agentic workflow will increase more than fivefold through 2028. Press release.gartner.com
48Anthropic. (2026). Pricing. Claude Platform documentation.platform.claude.com
AI Benefit Realisation · Parts 1 to 4
Coming next in the series
Two parts still to come.
Part 5
Stage-gates for AI pilot funding
Gates with kill criteria written in advance, decision rights, and a template for the stop decision.
Part 6
Reporting AI benefits to the board
Realised against forecast, net of run cost, with the confidence of each figure stated. A one-page report that survives an audit committee.
Parts 5 and 6 are provisional and may be renamed before they are written.
AI Benefit Realisation · Parts 1 to 4
AI Benefit Realisation · Parts 1 to 4
Thank you.
Read the full series at terencekok.com/blog/series/ai-benefit-realisation/
A gate that stops a weak pilot early is the cheapest control an AI programme will ever buy.
Read the series