Listen to this article
Jump to a section
In this article
Executive Summary
In September 2018, Jeff Bezos told an Air Force audience that Amazon’s culture rests on four things: customer obsession, long-term thinking, eagerness to invent, and professional pride in operational excellence. He singled out the last one, because “90 percent of what all of you do, nobody ever sees.” Nobody inspects it and nobody gives credit for it, so only your own pride keeps the standard high. That idea matters more in AI than almost anywhere else. A great many AI products are a screen and a prompt placed over a model someone else built. They demo well and are easy to copy. Google’s engineers wrote in 2015 that a mature machine learning system may be at most 5 percent model code and at least 95 percent everything around it. That surrounding work is the system. It decides whether the product can be trusted, defended and kept. I would like more builders to aim for it.
90%
of what people do at work that “nobody ever sees”, in Jeff Bezos’s words to the Air Force Association, September 2018
5%
at most of a mature machine learning system that is model code, with at least 95% the code around it, Sculley et al., Google, NeurIPS 2015
95%
of companies in MIT NANDA’s 2025 dataset reporting no measurable P&L impact from generative AI, with tools that do not learn from workflows named as the core problem
C$812
Air Canada was ordered to pay after its website chatbot told a grieving customer about a refund the airline’s policy did not allow, February 2024
Core conclusions
- The model is the smallest part of a working AI system. Data, evaluation, permissions, records, fallbacks and monitoring make up the rest, and they are where trust comes from.
- A wrapper is a fine way to test an idea and a weak place to stop. Anyone with the same model can copy it, and when it goes wrong there is nothing underneath to explain or contain the failure.
- The work nobody sees will only be done well if the people building it are proud of it. Leaders should ask about that work, review it and give it credit, because users never will.
The argument, in ten slides
Save it, or send it to the people building AI in your organisation.
What Bezos said about the work nobody sees
On 19 September 2018, Jeff Bezos sat down with retired General Larry Spencer at the Air Force Association’s Air, Space and Cyber Conference in Maryland. Spencer asked him what kept him awake at night. Bezos said he worried about Amazon losing its culture, and then described that culture in four parts:
“Customer obsession instead of competitor obsession; willingness to think long term, with a longer investment horizon than most of our peers; eagerness to invent, which of course goes hand-in-hand with failure; and then, finally, taking professional pride in operational excellence.”
He spent the most time on the last one:
“The professional pride part is so important, because 90 percent of what all of you do, nobody ever sees. It’s something that’s inside the work. People see something finished, but they don’t see what’s inside it. Only you see that while you’re doing the work. Nobody’s going to be inspecting it. You’re not going to get any credit for it. The only thing that’s going to make you have high standards on that piece of the work that nobody’s ever going to see is your own professional pride in operational excellence.”
He added that Amazon, AWS and Blue Origin use the same four attributes. I have returned to this passage many times this year, because it describes the choice in front of almost everyone building with AI right now.
What a wrapper is
A wrapper is an AI product whose main contribution is the layer the user touches. There is a screen, a prompt that someone has tuned, perhaps a template or two, and behind it a model built and run by another company. Many useful tools start this way, and I have no quarrel with that. A wrapper is the fastest way to learn whether an idea is worth anything.
The trouble starts when the wrapper is also where the work stops. Three things follow.
First, it is easy to copy. Anyone with access to the same model can rebuild the screen and the prompt in a weekend, and the model provider can ship the same feature to every one of its users at once. I wrote about this in why new AI products struggle to stand out.
Second, it cannot account for itself. When a wrapper gives a wrong answer, there is usually no record of what it was given, what it retrieved, which version of the prompt was live, or why it answered the way it did. The builder can only try again and hope.
Third, it does not improve. Each conversation starts from nothing. The wrapper does not learn which answers were corrected, which documents were out of date, or where users gave up.
Each of these is a problem with the 90 percent that nobody sees. The screen can be beautiful and the product can still be hollow.
Why the model is the small part
In 2015, a team of Google engineers led by D. Sculley published a paper called Hidden Technical Debt in Machine Learning Systems. Its best-known figure is a set of boxes. A tiny black box in the middle is labelled “ML Code”. Around it sit far larger boxes: configuration, data collection, feature extraction, data verification, machine resource management, analysis tools, process management tools, serving infrastructure and monitoring. The caption reads: “Only a small fraction of real-world ML systems is composed of the ML code… The required surrounding infrastructure is vast and complex.”
The text goes further. A mature system, the authors wrote, “might end up being (at most) 5% machine learning code and (at least) 95% glue code.”
The model is the small box in the middle.
Redrawn after Figure 1 of Sculley et al., Hidden Technical Debt in Machine Learning Systems, Google, NeurIPS 2015. Box sizes are schematic. The 5% and 95% figures are the paper's own estimate for a mature system.
That paper was written before large language models were a product category, and the proportions in any given system will differ. The shape has held. When a company buys access to a foundation model today, it is buying the small black box. Everything that makes the box safe and useful inside a particular business still has to be built, and most of it is invisible to the person typing a question.
There is a nuance worth keeping. The same paper recommends wrapping black-box packages in common interfaces, so that the rest of the system does not depend on the quirks of one package. Wrapping a model as an engineering technique is good practice. Treating the wrap as the whole product is the mistake.
What happens when the inside is missing
On 11 November 2022, the day their grandmother died, Jake Moffatt asked the chatbot on Air Canada’s website about bereavement fares. The chatbot said they could book now and apply for the reduced bereavement rate within 90 days of the ticket being issued. Moffatt booked two full-price flights on that basis. The airline’s own bereavement page said the policy did not apply once travel was complete, and Air Canada refused the refund.
At the British Columbia Civil Resolution Tribunal, Air Canada argued that the chatbot was responsible for its own actions. The tribunal member called this “a remarkable submission”. “It should be obvious to Air Canada that it is responsible for all the information on its website,” he wrote. “It makes no difference whether the information comes from a static page or a chatbot.” He found the airline had not taken reasonable care to make sure its chatbot was accurate. In February 2024 Air Canada was ordered to pay C$812.02: C$650.88 in damages, plus interest and tribunal fees.
The sum was small. The principle was not: the company that puts an AI in front of its customers owns what that AI says. A chatbot that can state a policy needs something underneath it that ties its answers to the real policy, records what it said and to whom, and hands the conversation to a person when it is unsure. Those are invisible parts. In this case, as far as the public record shows, they were not there.
The pattern repeats at a larger scale. MIT’s Project NANDA reported in 2025 that 95 percent of the companies in its dataset saw no measurable effect on profit and loss from their generative AI efforts. The report blamed a “learning gap” more than model quality: generic tools that work well for an individual stall inside a company because they do not learn from, or adapt to, the way the work is done. The study’s methods have been questioned and I would not lean hard on the exact figure. The diagnosis matches what I see. The model was rarely what was missing. What was missing was everything around it.
What sits inside a good AI system
When I build an AI system now, whether for a client or as a working concept on this site, I keep a list of what it needs underneath the screen. Users will never see most of it. All of it decides whether I can stand behind the system.
| Part | What it does | Who notices when it is missing |
|---|---|---|
| Grounding in real data | Ties answers to the documents, records and policies the organisation holds, with the source shown | The customer who is told something that is not true |
| Evaluation set | A fixed set of test questions with known good answers, run after every change to the prompt, model or data | The team that ships a change and finds out a week later it made things worse |
| Permissions | Checks who is asking and what they are allowed to see or do before the model acts | The auditor, after someone sees data they should not have |
| A record of every call | Stores what went in, what was retrieved, which model and prompt version ran, and what came out | The person asked to explain a decision six months later |
| Fallback and hand-off | Knows when to say it is unsure and pass the case to a person | The customer stuck in a loop with no way out |
| Monitoring | Watches error rates, refusals, cost and drift, and raises an alert before users complain | The operations team, at 2am |
| Cost metering | Counts tokens and spend per user, feature and call | Finance, when the invoice arrives |
| Versioning and rollback | Lets you return to yesterday’s prompt, model or index in minutes | Everyone, on the day a model update changes behaviour |
My own working list, drawn from the systems I build and review.
The same screen can sit on very different foundations.
The system: the same screen and model, with the eight parts users never see built in underneath. This is the part that earns trust.
The eight parts are my own working list, from the table above. Schematic.
None of these parts are new ideas. Most of them are ordinary engineering discipline applied to a component that happens to be probabilistic. They are also exactly the work that never appears in a demo and rarely appears in a launch announcement.
Experiments and operational excellence
In the same 2018 conversation, Bezos made a distinction that I think every AI team should adopt. Big organisations, he said, “start to confuse experimentation with operational excellence.” Amazon had built more than 150 large fulfilment centres by then. “If we build the 151st fulfillment center and screw it up, that’s just a failure. That’s not the kind of failure we’re seeking.”
AI work needs both kinds of effort, and it needs people to know which one they are doing. A pilot with ten friendly users is an experiment. A wrapper is a perfectly good way to run it, and if it fails, the organisation has learned something cheaply. Once customers, staff or regulators depend on the system, it has become a fulfilment centre. A failure there is just a failure, and the invisible parts in the table above are what prevent it.
The mistake I see most often is a successful experiment being promoted straight into production with nothing added underneath. The demo worked, so the demo shipped. I described what usually happens next in why AI demos work and production systems fail.
How to tell a system from a wrapper
You do not need to read the code to tell the difference. Five questions will usually do it, and they work as well in a vendor meeting as in a review of your own team’s work.
If the model provider changed tomorrow, what would break? A system has its own data, tests and records, and can move to another model with work. A wrapper is the provider’s model with a different front door.
Can you show me what it did last Tuesday, and why? A system can retrieve the exact input, sources, model version and output for any past answer. A wrapper usually cannot.
How do you know the last change made it better? A system has an evaluation set and a before-and-after score. A wrapper has a feeling.
What happens when it is wrong? A system has a defined path: a confidence check, a hand-off to a person, a correction that feeds back into the tests. A wrapper has an apology.
Who finds out first when it starts to fail? In a system, a monitor raises an alert and someone on the team is told. With a wrapper, the first to know is usually a customer.
Where to start
If you build AI, pick the one invisible part from the table that your current project lacks most, and build it properly this month. For most teams that is the evaluation set or the record of every call. Nobody will ask you for it. You will know it is there, and the first time something goes wrong you will be very glad of it.
If you lead people who build AI, ask about the 90 percent. Bezos said that teaching culture is about repetition, and that leaders have succeeded when people roll their eyes because they have heard it so often. Ask to see the evaluation results in every review. Ask how the system would explain a decision from last month. When someone builds a good rollback or a careful permission check, say so in front of the team. Users will never thank them for it, so their leaders should.
The screen is what people see first. The system underneath is what they come back to, and what keeps them safe when something goes wrong. It is worth building well, and worth being proud of.
Evidence & Methodology
The Bezos quotes, the Google paper and the Air Canada ruling are on the public record and cited below. The MIT figure is widely reported and its method has been questioned. The table of system parts and the five questions are my own. Here is which is which.
| Claim | Source | Grade |
|---|---|---|
| Bezos described Amazon’s culture as four attributes and said “90 percent of what all of you do, nobody ever sees”; Amazon, AWS and Blue Origin use the same four | Transcript of his talk with Gen. Larry Spencer at the Air Force Association’s Air, Space and Cyber Conference, 19 September 2018, published by GeekWire | Measured, primary transcript |
| Bezos said big organisations confuse experimentation with operational excellence, citing the 151st fulfilment centre | Same transcript | Measured, primary transcript |
| Only a small fraction of real-world ML systems is ML code; a mature system might be at most 5% ML code and at least 95% glue code | Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015, Figure 1 and section 5 | Measured, written before LLM products |
| Air Canada was held responsible for its chatbot’s wrong statement of the bereavement policy and ordered to pay C$812.02, of which C$650.88 was damages and the rest interest and fees; the tribunal called the “separate entity” argument remarkable | Moffatt v. Air Canada, 2024 BCCRT 149, 14 February 2024, read in the original decision | Measured |
| 95% of companies in the dataset saw no measurable P&L impact from generative AI; a learning gap in tools and organisations was the core cause | MIT Project NANDA, The GenAI Divide: State of AI in Business 2025, as reported by Fortune; based on 150 interviews, a survey of 350 employees and 300 public deployments | Reported, method disputed |
| Eight invisible parts of a good AI system | My own working list from systems I build and review | My framework |
| Five questions that separate a system from a wrapper | My own review practice, not a validated assessment | My framework |
| Successful experiments are often promoted into production with nothing added underneath | My own observation in client and review work | My call |
Sources
- Boyle, A. (2018, September 19). Transcript: ‘Chief slowdown officer’ Jeff Bezos shares Amazon management tips. GeekWire.
- Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems 28.
- Civil Resolution Tribunal of British Columbia. (2024, February 14). Moffatt v. Air Canada, 2024 BCCRT 149.
- Estrada, S. (2025, August 18). MIT report: 95% of generative AI pilots at companies are failing. Fortune.
Why AI Demos Work and Production Systems Fail
What has to change between a demo that impresses and a system people can rely on.
The Productivity Gap: Why Companies Must Fix Their Systems Before Scaling AI
Why pilots stall when the systems around the model were never built for it.
Why New AI Products Struggle to Stand Out
Why products built on the same models and prompts converge, and what lasts.
If your organisation is moving AI from pilots into systems people depend on, that is the work my executive programme is built around.
Was this useful?
