Why Large Enterprises Are Deploying AI at Scale but Measuring It Incorrectly

The measurement gap, in ten slides

Save it, share it, or send it to whoever signs off your next AI scale-up budget.

Swipe to preview · 10 slidesDownload the PDF

Across Asia-Pacific, large enterprises are investing heavily in AI. The infrastructure exists, the vendor contracts are live, and the models are in production. The measurement frameworks, however, are still tied to the wrong layer of the technology stack.

The Measurement Problem

Enterprise AI programmes in Asia are speeding up on most measures. Capital spending on AI infrastructure across Southeast Asia, Greater China, and the Indian subcontinent reached record levels in 2025. Deployment timelines have shortened as cloud-native tools have matured. The models are running. What most programmes still cannot answer well is whether the models are producing the outcomes that justified the investment.

The gap is structural. Most organisations took their AI measurement habits from the teams that buy technology and test software. Those teams judge systems on technical measures: model accuracy, F1 scores, precision-recall trade-offs, latency benchmarks, and system uptime. These are valid engineering metrics. They are not, however, business performance metrics. An organisation that treats the two as the same believes it is succeeding when its operations have not materially changed.

A language model that returns accurate outputs 94% of the time is not a business result. It is a capability specification. The business results are less manual processing time, more consistent decisions, a measurable change in cost per transaction, or a shift in customer resolution rates you can show. These are different measurement problems, and each needs its own measurement infrastructure.

Model accuracy tells an organisation what its AI can do. Operational impact metrics tell it what the AI has done.

Model Performance Metrics vs Business Performance Baselines

The difference between model performance metrics and business performance baselines is not a matter of preference. It is a matter of which layer you are measuring. Mixing them up does more than make reporting imprecise. It leads to decisions that optimise the wrong variable, and it hides real programme risk.

Model performance metrics operate at the technical layer. These metrics describe how the model itself behaves, on its own or under controlled test conditions. Common examples include:

• Accuracy and F1 score: share of correct predictions on labelled test sets
• Precision and recall: the trade-off between false positives and false negatives in classification tasks
• Inference latency: time taken to return an output under set load conditions
• Model drift indicators: how far live inputs have moved from the training data over time, measured statistically
• Token throughput and cost per inference: what it costs to run the model at scale

These metrics are essential for engineering teams who manage model reliability and infrastructure cost. They fall short as programme-level reporting. They say nothing about whether anyone uses the model’s outputs, whether those outputs improve decisions, or whether the business process they were meant to augment has changed at all.

Business performance baselines operate at the operational layer. A business performance baseline records the state of a measurable operational process before AI is deployed. It is set against specific workflows, teams, or transaction types, not against the model itself. Common baseline metrics include:

• Processing time per transaction: average and 90th-percentile time for a defined workflow step before AI is added
• Decision consistency rate: share of like cases that get like outcomes when only people handle them
• Error or exception rate: how often downstream corrections are needed per unit of output
• Cost per unit of output: fully loaded cost including labour, rework, and escalation
• Throughput capacity: number of cases processed per unit of time at baseline staffing levels

Without these baselines in place before deployment, you cannot attribute any later change in performance to AI. This is not a technical limitation. It is a programme governance failure. Baselines must be defined, measured, and locked before the model enters the production workflow.

Why This Pattern Is Prevalent Across Enterprise Deployments

Large Asian enterprises tend to measure the model for several structural reasons, and those reasons are broadly the same across the region.

First, in many large enterprises across Southeast Asia and Greater China, the technology function sponsors the AI programme. Technology leaders are rewarded for showing that the technology works. Showing the other half, that the technology has changed something in the business, needs sponsorship from business unit leaders, and they are often not part of the programme from the start.

Second, the way vendors are engaged adds to the problem. Most enterprise AI vendors in the region are evaluated and contracted on model performance benchmarks. Once a vendor delivers a model that meets the accuracy figure in its contract, the job is technically done, whether or not the enterprise has seen any operational change. The measurement gap is therefore built into procurement.

Third, deployment has moved faster than measurement infrastructure. Organisations have gone from proof of concept to production in short timeframes. In many cases the tools that capture pre-deployment baselines go in at the same time as the model. That makes a meaningful before-and-after comparison structurally impossible.

“The instrumentation required to capture pre-deployment baselines must be installed before the model. Concurrent installation makes a before-and-after comparison structurally impossible.”

The Consequences of Measuring the Wrong Thing

Measuring only the model has practical costs, and they show up in several places over the life of a programme.

At the reporting layer, boards and executive committees are told the AI programme is performing well, while the operational metrics that would show its real impact go uncaptured. Scale-up budgets get approved for programmes that did not show an operational return at the pilot stage.

At the risk layer, model drift (the gradual decline in model performance as the distribution of live data shifts away from the training data) is well understood. If no one is watching operations as well, a model can get materially worse before anyone sees the effect on the business. By the time the decline shows up in exception rates or manual correction volumes, the lag is significant.

At the governance layer, organisations subject to audit and regulatory oversight face growing scrutiny of how AI makes decisions. This applies above all to financial institutions and infrastructure operators, under MAS guidelines in Singapore or sectoral rules across ASEAN. To show outcomes that are consistent, auditable, and bounded, they need operational evidence, not model performance certificates.

Integrating Both Measurement Layers Corrects the Gap

Closing the measurement gap does not mean replacing existing model monitoring infrastructure. It means extending measurement to the operational layer and setting up governance that ties both layers into one programme reporting structure.

1. Define operational baselines before deployment. Identify the specific workflows the AI system will augment or automate. Measure the current state on the indicators listed above. Lock the baseline values as programme artefacts, under version control and open to programme governance.
2. Establish operational KPIs with explicit attribution rules. Decide which operational changes will count as the result of AI deployment, over what time window, and against what counterfactual controls. Attribution without set rules produces numbers people will dispute.
3. Instrument the workflow. Put observability in place at the point where model outputs enter operational processes. Track whether human operators accept, override, correct, or escalate those outputs.
4. Connect model monitoring to operational alerting. When model drift indicators cross set thresholds, review the operational metrics in the affected workflows. Do not treat model and operational monitoring as separate reporting streams.
5. Report operational impact to executive governance on a defined cadence. Model performance metrics belong in engineering dashboards. Operational impact metrics belong in board and investment committee reporting.

Conclusion

Across Asia, enterprise AI works technically, and the money behind it is committed. The measurement infrastructure, however, has not kept up with the pace of deployment. The result is a growing group of large enterprises that can show model performance but still cannot show operational return.

This is not a problem of technology. The instruments needed to capture operational baselines and measure impact after deployment are available and well understood. It is a problem of programme design and of doing governance steps in the right order: specifically, the failure to make measurement infrastructure a precondition for deployment.

Organisations that fix this sequencing error will be able to make scale-up decisions they can defend. They will meet the growing regulatory scrutiny of AI systems, and they will be able to tell real operational improvement apart from technical compliance. Those that do not will keep approving investments on metrics that cannot answer the question their boards are asking.

I built the Measurement-Before-Prediction Architecture as a structured way to design measurement infrastructure before deployment begins, which is the order this article argues for. If you’d rather drop headcount-reduction metrics entirely, the Return on Employee framework is my alternative for capturing genuine operational impact.