Listen to this article
Executive Summary
The question has shifted from “can AI do this” to “can this run at scale, on a budget, without falling over.” Cost now scales with adoption, not with the value being created.
Core conclusions
- Two problems are surfacing at once: routing every request to a frontier model makes cost track adoption rather than value, and the open-source ecosystem is straining under AI-generated contributions maintainers can’t keep up with.
- Five architectural moves fix it: route by complexity instead of defaulting to the frontier model, build automatic failover, centralise governance, distil frontier output into smaller fine-tuned models, and ask whether a model is needed at all.
- The most-skipped move is the last one. Once a use case leaves proof-of-concept, the real question is whether a lookup table or rules engine would do the job without a model at all, not which model to pick.
The five moves, in ten slides
Save it, share it, or send it to whoever owns the infrastructure bill.
For the past couple of years, the story was speed: ship the pilot, plug in the frontier model, show the demo. That phase is ending. What’s left is a much less glamorous question: can this actually run at scale, on a budget, without falling over?
Two problems are surfacing at the same time.
The Bill Is Coming Due
Routing every request to a top-tier model sounds fine until usage grows. Then cost scales with adoption, not with the value being created. Add the lock-in of building everything around a single provider’s API (pricing changes, outages, painful migrations), plus the less visible costs: context management, caching, infrastructure, keeping models current. The real total cost of ownership looks very different from the number in the original business case.
The Open-Source Ecosystem Is Straining Under Its Own Success
AI coding agents are now submitting pull requests faster than maintainers can review them. Some of that code carries invented dependencies or security issues that slip past normal review. The result: maintainer burnout, and in some cases, projects tightening their contribution rules or shutting the door altogether.
Neither problem means AI adoption should slow down. It means the architecture has to grow up.
I go deeper on this in the video below:
Five Moves for Teams Past the Pilot Stage
- Stop sending every request to the biggest model. Put a proxy layer between your applications and the models themselves. Simple, routine requests go to smaller, cheaper models. Only genuinely complex reasoning gets sent to the frontier model. This alone reshapes the cost curve.
- Build in failover. If one provider has an outage or a latency spike, traffic should shift automatically to another model: no manual intervention, no downtime.
- Centralise governance. One layer for rate limits, cost tracking, and data controls, instead of every team managing this on its own.
- Distil, don’t just call. Use frontier models to generate high-quality training data, then fine-tune smaller open-source models on that data for your specific use case. Done well, these smaller models match frontier performance on narrow tasks, for a fraction of the cost, with lower latency, and fewer data residency headaches.
- Ask whether you need a model at all. This is the one most teams skip. Use AI to prototype. Don’t assume it belongs in production. Once a use case moves from proof-of-concept to implementation, the right question isn’t “which model should we use,” it’s “do we need a model at all.” If the logic can be handled by a lookup table, a rules engine, or any other deterministic method, that should be the default. AI earns its place in production through genuine necessity, not habit.
The organisations getting real ROI from AI aren’t the ones with the flashiest demo. They’re the ones treating model choice as an infrastructure decision, and willing to ask, at every stage, whether AI is the right tool at all.
Free tool
AI Build vs. Buy vs. Partner
Applies seven criteria, including data sensitivity, internal capability, and maintenance appetite, to the same question this article ends on: whether AI is the right tool at all.
Free tool
AI ROI Calculator
Model the cost curve before you commit a use case to a frontier model in production, so cost scales with value instead of adoption.
Was this useful?