What Keeps You Awake at Night About AI?

Listen to this article

What Keeps You Awake at Night About AI?

0:00

The five failure modes, in ten slides

Save it, share it, or send it to whoever runs your control room.

Swipe to preview · 10 slidesDownload the PDF

People assume that because I run AI and smart city programmes across Asia and the Middle East, I must lie awake at night worrying about rogue superintelligence.

I do not.

The models mostly do what they are told.

What actually keeps me awake is everything wrapped around the models: the messy data, the missing governance, the invisible failure modes, and the very human habit of trusting shiny dashboards a bit too quickly.

Let me explain.

1. AI That Fails Quietly

If an AI system burst into flames when it went wrong, life would be simple.

Instead, it smiles, outputs a neat confidence score, and quietly nudges a critical system in the wrong direction.

In operational environments (water, power, transport, buildings) models drift over time. Sensors get slightly miscalibrated. Operating conditions change. The model does not send an apology email. It just keeps making slightly worse decisions.

One day, a “smart” system for optimising water pressure starts chasing ghosts in the data. Nobody gets an alert because nobody set proper guardrails. The AI looks confident. The dashboards look green. The pipes do not.

The punchline: the AI is behaving as designed. The governance is not.

2. The “Magic AI on Top of a Mess” Strategy

Across the region, governments and operators are investing heavily in AI, digital twins, and smart city platforms.

Yet behind many glossy presentations sits the same pattern:

• Ten agencies.
• Twenty legacy systems.
• Hundreds of spreadsheets.
• Zero shared data model.

Then someone says, “No problem. Let’s put AI on top of it.”

It works beautifully in a demo. The model summarises reports. It generates insights. Everyone nods.

Then you try to move from pilot to production and discover that half the data is incomplete, a third is duplicated, nobody owns the pipelines, and the “single source of truth” is actually five different databases last reconciled during a different administration.

At that point, it is not AI holding you back. It is basic data hygiene.

3. Hallucinations Are Not Cute in Government

In a consumer app, an AI hallucination is mildly annoying.

In government and critical infrastructure, it is the difference between:

• A fun wrong answer.
• A wrong benefit decision, a wrong fine, or a wrong operational action.

Treating hallucination as a “bug the vendor will eventually fix” is wishful thinking. These systems are probabilistic by design. If you deploy them without clear boundaries, verification steps, and humans who are trained (and allowed) to say “this looks wrong”, then you have not bought a tool. You have bought a governance problem.

What worries me is not that AI sometimes makes things up.

It is that people assume “the system must be right” because it has an official logo on the interface.

4. Autonomous Loops With No Off Switch

Somewhere in a control room, someone says, “For now, it’s just decision support; the human is still in the loop.”

Six months later:

• Humans are too busy to challenge every recommendation.
• The AI system is accurate most of the time.
• Nobody formally updates the risk classification.

Congratulations: the AI has quietly moved from decision support to de facto autonomy.

Now imagine that system is adjusting traffic signals, or balancing loads on an energy grid, or tuning building systems across a campus.

If something goes wrong, the most important question is suddenly very simple:

“Where is the big red OFF button?”

Too often, it does not exist in a tested, practical way. There is a policy document. There is a diagram. But there is no simple, rehearsed, “this is what you do at 3am when the system goes weird” procedure. This is exactly the design gap I address in Redesigning Oversight Architectures: decision support quietly becoming de facto autonomy is a governance failure, not a technology one.

5. The Great Accountability Hide‑and‑Seek

Ask in any large AI programme:

“If this system causes harm, who is actually accountable?”

You will hear variations of:

• “The vendor follows the contract.”
• “The system only makes recommendations.”
• “The final decision is the human’s.”
• “We have a committee.”

In other words: everyone is involved; nobody is responsible.

Vendors point to the terms and conditions. Agencies point to the vendor. Operators point to “the system”. The system, of course, is not available for comment.

What I worry about is not a spectacular failure. It is the slow accumulation of small, unowned decisions, each one made by a system that nobody quite feels fully responsible for, and which no one can easily switch off.

6. What I Actually Want to See

Despite all this, I am not pessimistic about AI. I am pessimistic about AI without discipline.

Things that genuinely help me sleep:

• Clean, documented data pipelines that someone actually owns.
• A simple map of “these are all the AI systems we run, and this person is accountable for each of them”. This is close to what I mean by the four dashboards every Chief AI Officer should actually be running.
• Clear rules for which systems must always have a human in the loop.
• Monitoring that looks at how the AI behaves in production, not just in the lab.
• A tested, boring, non‑dramatic way to shut systems down or roll them back.

None of this is futuristic. It is basic engineering, operations, and governance, applied seriously to AI instead of treated as an afterthought.

So, What Keeps Me Awake?

Not superintelligence.

Not robots taking over cities.

What keeps me awake is the gap between:

• How confident organisations sound when they talk about AI.
• How prepared they actually are to operate it safely, fairly, and visibly in the real world.

The good news is that the gap is fixable. The bad news is that it requires less hype and more unglamorous work. And that, more than the AI itself, is what keeps many of us in this field awake at night.