Executive Summary
Standard AI models are mathematically forced to guess, even on data they’ve never seen, and that overconfidence is an operational liability in embedded infrastructure systems. The fix is architecting a “reject option” so the system abstains instead of guessing when uncertainty crosses a safety threshold.
Core conclusions
- Softmax-based models can’t output “unknown.” They’ll assign high confidence to inputs that share superficial features with their training classes, disguising a wild guess as a confident fact.
- Distinguishing aleatoric uncertainty (noise) from epistemic uncertainty (ignorance) matters because the correct safety response differs for each.
- Conformal Prediction gives a mathematical coverage guarantee and lets a system abstain and route to a human when it’s genuinely uncertain. Leaders should ask their data teams for a rejection rate, not just accuracy or F1.
The reject option, in ten slides
Save it, share it, or send it to whoever owns your model’s failure modes.
A prediction error is not merely a statistical anomaly. It is an operational liability. A recommendation engine that misclassifies a movie causes mild annoyance; a computer vision system that misclassifies a red traffic signal as green during a monsoon causes a fatality.
As we scale AI-based embedded intelligence across water grids, transport networks, and energy systems, we face a critical architectural challenge. Standard deep learning models are notoriously overconfident. They will predict a class with 99% probability even when the input data lies entirely outside their training distribution.
For leaders delivering resilient infrastructure, the most valuable output an AI system can generate is not a prediction, but a refusal to predict. We must move from systems that guess to systems that know when they do not know.
The most valuable output an AI system can generate is not a prediction, but a refusal to predict.
The Illusion of Confidence
Most commercial AI models rely on a “softmax” output layer, which forces the system to distribute probability across a fixed set of classes. If a traffic camera trained on cars and buses sees a fallen tree, it does not output “unknown object.” It outputs “bus: 85%,” simply because the tree shares some textural features with a bus chassis.
Imagine a student taking a multiple-choice exam where they must select an answer for every question. Even if they encounter a question written in a language they don’t speak, they cannot leave it blank. They must bubble in A, B, C, or D. If they pick “C” just because it looks vaguely familiar, the grading machine assumes they are confident in that answer. Standard AI works the same way: it is mathematically forced to make a choice, often masquerading a wild guess as a confident fact.
Engineering the “Reject Option”
Reliable AI requires differentiating between two types of uncertainty:
- Aleatoric Uncertainty: Randomness inherent in the data (e.g., sensor noise during a sandstorm). More data cannot fix this; the system must learn to operate within these noise margins.
- Epistemic Uncertainty: Ignorance caused by a lack of training data (e.g., a new type of pipe material the model has never seen). This is solvable with better data, but the model must first identify its own ignorance.
Think of this distinction as driving a car. Aleatoric uncertainty is like driving through heavy rain: your perception is obscured by the environment, and no amount of training will make the rain disappear. Epistemic uncertainty is like driving in a foreign country and seeing a road sign you have never learned before. The weather is clear, but you lack the knowledge to interpret the sign. A smart system needs to know whether it is “blinded by rain” (noise) or simply “uneducated” (lack of training), because the safety response for each is different.
The Technical Solution: Conformal Prediction
We address this by architecting a “Reject Option,” a mechanism that allows the model to abstain from decision-making when uncertainty crosses a safety threshold. While Bayesian Neural Networks offer one path, a more robust approach for industrial applications is Conformal Prediction.
Unlike standard confidence scores, Conformal Prediction provides a mathematical guarantee of coverage. Instead of predicting a single class, the system outputs a set of probable classes (e.g., “Traffic Light: [Red, Amber]”) with a user-defined error rate (e.g., 1%). If the set is empty or contains contradictory classes, the system flags the input as “out-of-distribution” and abstains.
Standard AI operates like a doctor who is forced to give you a single, specific diagnosis instantly (“You definitely have the flu”) even if the symptoms are vague. Conformal Prediction is like a careful doctor who says, “Based on these symptoms, it could be the flu, or it could be a bacterial infection. I cannot pinpoint it yet.” By providing a set of possibilities rather than one risky guess, the system avoids a wrong call and routes the decision to a human operator.
Operational ROI: The Value of Abstention
Implementing abstention logic fundamentally alters the ROI of digital twin and IoT programmes.
1. Water Network Resilience. Consider acoustic sensors deployed to detect leaks in a city’s ageing water network. A standard model might achieve 92% accuracy. However, the 8% false positive rate results in costly excavation teams digging up dry roads. By implementing an abstention layer, the system can autonomously filter low-confidence alerts:
- High Confidence Leak: Auto-dispatch repair crew.
- High Confidence Normal: No action.
- Uncertain/Abstain: Route raw audio to a human analyst.
Result: Alert fatigue drops and excavation costs are only incurred when certainty is high.
2. Transport Safety. In autonomous traffic management, visual sensors must handle extreme weather. When heavy rain degrades sensor fidelity (high aleatoric uncertainty), a standard model might guess traffic flow to optimise signal timing. A robust system recognises the data degradation, abstains from AI optimisation, and reverts to deterministic failsafe timing logic until conditions improve. Designing that fallback layer well is the same oversight-architecture problem I address in Redesigning Oversight Architectures.
The Leadership Imperative
For those governing integrated ecosystems, the metric of success must shift. Do not ask your data science teams solely for “Accuracy” or “F1 Score.” Ask them for the rejection rate.
- How does the model behave when it sees data it wasn’t trained on?
- Is there a fallback protocol for high-uncertainty predictions?
- Are we measuring the cost of a false positive versus the cost of a manual review? The rejection rate belongs alongside the other dashboards a Chief AI Officer should be running.
True intelligence recognises its own limitations. When our AI system admits “I don’t know,” it is not failing. It is demonstrating the highest level of reliability.
Free tool
Model Performance & Health Dashboard
The rejection rate this piece argues for is a monitoring metric, not a one-off score; track it alongside drift, latency, and rollback history for every deployed model.