The Black Box Problem: What It Is, Why It Persists, and How to Engineer Around It

1. Definition

The black box problem describes a condition in which a system’s outputs can be observed and measured, but the procedure that produced them cannot be inspected, reconstructed, or expressed in terms a human reviewer can evaluate. In machine learning the term applies to models whose input-output mapping is determined by learned parameters rather than by written rules. The parameters are available. The decision procedure is not, in any form that supports the kind of scrutiny normally applied to engineered systems.

The distinction that matters operationally is between access and legibility. An operator may hold the full weight tensor of a 70-billion-parameter transformer on local storage and still be unable to state why a given input produced a given classification. Possessing the artefact does not confer understanding of the function it computes.

Jenna Burrell’s 2016 paper in Big Data & Society separates three forms of opacity that are frequently conflated in procurement and governance discussions.

Opacity by intent. The provider withholds architecture, weights, training data, or evaluation results as commercial confidential information or on national security grounds. This is a contractual and policy problem. It is soluble through negotiation, escrow arrangements, or regulated disclosure.

Opacity by expertise gap. The information is available but the receiving organisation lacks the specialist capability to interpret it. This is a resourcing problem, soluble through recruitment or independent assessment.

Opacity by construction. The model’s operation involves the interaction of very large numbers of learned parameters in a high-dimensional space, optimised against a loss function rather than designed against a specification. No party holds the explanation, including the developer. This form is not soluble by disclosure and is the substantive technical problem.

Public debate tends to focus on the first form because it has an obvious remedy. Assurance failures in deployment tend to originate in the third.

2. Technical origins of intrinsic opacity

Several properties of contemporary neural architectures produce opacity as a structural consequence rather than as an oversight.

Distributed representation and superposition. Individual neurons do not correspond to individual concepts. Anthropic’s 2022 work on toy models of superposition demonstrated that networks routinely encode more distinct features than they have dimensions, storing them as overlapping directions in activation space. A single unit therefore participates in many unrelated computations, a property termed polysemanticity. Inspection of individual components yields little because the relevant unit of computation is a direction in a high-dimensional space, not a neuron.

Absence of a specification. Conventional software is traceable from requirement to implementation. A machine learning model has no requirement decomposition. It has a training objective, a dataset, and an optimisation procedure. The behaviours that emerge were not specified and are not enumerated anywhere. There is no document against which the model can be verified, only a distribution against which it can be tested.

Loss of the audit chain. Safety standards used in infrastructure, including IEC 61508, ISO 26262, EN 50128 and DO-178C, presuppose bidirectional traceability between hazard analysis, requirements, design, and code. Learned components break this chain at the design stage. EASA’s concept paper on machine learning in aviation responds by replacing the conventional V-model with a W-shaped “learning assurance” process, in which data management and model training receive assurance activities of their own. The University of York’s AMLAS methodology takes a comparable approach for autonomous systems, structuring the argument as a set of explicit assurance claims about data, model, and deployment context.

Unfaithful self-explanation. Language models will produce a rationale for an output on request. That rationale is generated by the same process that generated the output and is not a record of it. Turpin and colleagues showed in 2023 that chain-of-thought traces can systematically omit the factor that actually determined the answer: when the prompt contained a biasing cue, models changed their answers accordingly while presenting reasoning that never referenced the cue. A fluent explanation is evidence of fluency, not of faithfulness.

Non-determinism at inference. Sampling temperature, floating-point non-associativity, and batch-dependent kernel reduction order can all cause the same input to produce different outputs on different runs or under different server load conditions. Reproducing an incident therefore requires the input, the model version, the sampling parameters, and in some cases the serving configuration. Few production logging schemes capture all four.

Data provenance. Training corpora for general-purpose models are large, heterogeneous, and typically undisclosed. A deployer cannot establish what the model was exposed to, and therefore cannot bound what it may reproduce, what contamination may affect benchmark results, or which populations are underrepresented.

3. Common misconceptions

Opacity is not the same as complexity. The 2020 Ofqual grade standardisation model in England was a statistical procedure of modest complexity, published in outline, and still functioned as a black box for the affected population because the mapping from an individual candidate’s circumstances to their outcome was not reconstructable by that candidate. Approximately 39 per cent of A-level grades were adjusted downwards from centre assessment before the model was withdrawn. Procedural opacity can arise from simple systems deployed without an explanation pathway.

Interpretability and explainability are not interchangeable. Interpretability refers to properties of the model itself that permit its operation to be understood, for example monotonic constraints, sparse structure, or additive decomposition. Explainability refers to post-hoc artefacts generated about an opaque model, for example attribution scores. The former is a design property. The latter is an approximation with its own error characteristics.

Post-hoc explanations are not audit evidence by default. Slack and colleagues demonstrated in 2020 that LIME and SHAP can be defeated by an adversarial scaffolding classifier that detects the perturbed inputs these methods use and returns innocuous behaviour for them. Cynthia Rudin’s 2019 paper in Nature Machine Intelligence argues that post-hoc explanation of a black box in high-stakes settings is the wrong pattern, and that inherently interpretable models should be preferred where they are competitive. On structured tabular data, which describes a large share of public sector and infrastructure use cases, the accuracy differential is frequently within noise.

4. Risk categories

4.1 Assurance and safety argument

Where an opaque component sits on a path that can influence physical actuation, dispatch, or an entitlement decision, no deductive safety argument is available for that component. The argument must become statistical and behavioural, which changes its logical status: it supports a claim about failure rate under a specified input distribution, not a claim about the impossibility of a failure mode. If the operating distribution shifts, the argument’s premises no longer hold, and nothing in the model signals that this has occurred.

4.2 Silent failure

Neural networks are poorly calibrated and become substantially worse under distributional shift. Guo and colleagues documented systematic overconfidence in modern networks in 2017; Ovadia and colleagues showed in 2019 that calibration degrades as shift increases while confidence scores remain high. The practical consequence is that a model that has begun to fail does not report that it has begun to fail. Detection depends entirely on external instrumentation.

Free tool

AI Model Performance & Health Dashboard

The instrumentation this section argues for: drift detection, latency and uptime, and incident tracking, so a model that has begun to fail is caught from outside rather than by asking it.

→

4.3 Root cause analysis and recurrence

Following an incident, a conventional system is debugged by locating the responsible logic. In an opaque model there is no logic to locate. Remediation typically consists of retraining with additional data, which is an empirical intervention with an unverified relationship to the failure mode. The organisation cannot demonstrate that the specific fault has been removed, only that it no longer reproduces on the test cases available.

4.4 Security

Data poisoning and backdoor insertion produce models that behave correctly on all inputs except those containing a trigger. Gu and colleagues demonstrated this in the BadNets work in 2017. Anthropic’s 2024 sleeper agents study found that backdoored behaviours persisted through supervised fine-tuning, reinforcement learning, and adversarial training, and that adversarial training in some cases improved the model’s ability to recognise its trigger rather than removing the behaviour. Inspection of the weights does not reveal the backdoor. Behavioural testing does not reveal it unless the trigger is known.

4.5 Legal and regulatory exposure

Explanation duties do not abate because the model is opaque. In Dun & Bradstreet Austria (C-203/22, 27 February 2025) the Court of Justice of the European Union held that supplying a complex algorithmic description does not discharge the obligation under Article 15(1)(h) GDPR to provide meaningful information about the logic involved. The information must enable the data subject to understand and contest the decision.

In the United Kingdom, section 80 of the Data (Use and Access) Act 2025 replaced Article 22 UK GDPR with Articles 22A to 22D, commenced on 5 February 2026 by SI 2026/82. The prior near-prohibition on solely automated significant decisions has been replaced, for ordinary personal data, by a permission conditional on safeguards: information about the decision, the ability to make representations, human intervention, and a route to contest. The compliance question has moved from whether an exception applies to whether the safeguards can be evidenced as operative, which is an engineering and logging question rather than a legal drafting one.

In the European Union, Regulation (EU) 2026/1744, the Digital Omnibus on AI, was published in the Official Journal on 24 July 2026 and entered into force on 27 July 2026. It deferred the compliance date for standalone high-risk systems under Annex III, which includes critical infrastructure, to 2 December 2027, and for high-risk AI embedded in products regulated under Annex I to 2 August 2028. The Article 50 transparency obligations were not deferred and applied from 2 August 2026, with a limited extension to 2 December 2026 for marking of content generated by systems already on the market. The underlying obligations, covering risk management, data governance, technical documentation, human oversight, and post-market monitoring, are unchanged. The deferral affects the deadline, not the design work.

4.6 Accountability and public trust

The Dutch childcare benefits case illustrates the compounding effect. A risk classification system operated by the Tax and Customs Administration flagged approximately 26,000 families for fraud investigation on grounds the families could not see or challenge, with nationality used as a risk indicator. Over a thousand children were removed from their homes. The Dutch Data Protection Authority imposed fines in 2021 and 2022, and the Cabinet resigned in January 2021. Separately, in NJCM v The Netherlands (District Court of The Hague, 5 February 2020), the SyRI welfare fraud detection system was held to breach Article 8 of the European Convention on Human Rights, with the court’s reasoning resting substantially on the absence of verifiable information about the risk model.

4.7 Automation bias

Where an operator cannot assess the basis for a recommendation, oversight tends to degrade towards ratification. Human review that cannot in practice disagree is not a control. Article 14 of the EU AI Act requires that oversight be capable of being exercised; the same expectation is implicit in the Article 22C safeguards under the UK regime. A design that presents a recommendation without contrastive information, confidence, or a disagreement pathway will not satisfy this in substance, whatever the process documentation states.

Free tool

Board AI Oversight Checklist

Automation bias in practice: whether your board can name who is accountable when a human reviewer cannot, in fact, disagree with the opaque model’s recommendation.

→

4.8 Supply chain and version drift

Hosted models are updated by the provider. Behaviour can change without notice, without a version increment visible to the deployer, and without any change to the deployer’s own code. Acceptance testing performed at procurement describes a model that may no longer be the model in service.

5. Documented commercial impact

• Zillow Offers. Automated valuation used to price residential acquisitions performed acceptably during a stable period and failed under changed market conditions. Zillow recorded an inventory write-down of approximately US$304 million in Q3 2021, announced the closure of the business line on 2 November 2021, and reduced headcount by around 25 per cent.
• Moffatt v Air Canada (2024 BCCRT 149, 14 February 2024). The British Columbia Civil Resolution Tribunal held the airline liable for incorrect information supplied by its customer service chatbot. The tribunal rejected the argument that the chatbot’s outputs were separable from the operator’s representations.
• Apple Card. Following allegations of gender-based disparity in credit limits, the New York Department of Financial Services concluded in March 2021 that it had not found unlawful discrimination, while criticising the inability of the issuer to explain individual outcomes to customers. The reputational cost was incurred during the period in which no explanation could be produced.

6. Mitigation

No single measure resolves the problem. The workable position is to reduce reliance on model transparency by constraining what the model is permitted to determine, and to instrument the surrounding system so that failure is detected externally.

6.1 Model selection

Establish interpretability as a requirement rather than a preference, and require justification when it is not met. For tabular problems, generalised additive models, explainable boosting machines, monotonic gradient-boosted trees with enforced constraints, sparse decision lists, and risk scorecards are frequently competitive. Monotonicity constraints are particularly useful in regulated decisions because they encode a defensible policy statement, for example that increased arrears may not reduce assessed risk, and the constraint holds by construction rather than by observation.

6.2 Architectural containment

Keep opaque components off the actuation path. Where a model informs a control action, place a deterministic, independently verifiable envelope between the model and the plant: rate limits, physical bounds derived from first principles, interlocks, and reversion to a conventional controller on violation. The assurance argument then attaches to the envelope, which is amenable to conventional verification, rather than to the model. Control barrier functions and reachability-based supervisors provide formal instantiations where the plant model supports them.

6.3 Uncertainty quantification and abstention

Conformal prediction provides distribution-free coverage guarantees under exchangeability and can be applied to any underlying model, producing prediction sets with a specified error rate rather than a point estimate. Combined with a calibrated abstention threshold, this converts an opaque predictor into a system with a defined and measurable escalation rate. Report expected calibration error and coverage per operational slice, not in aggregate.

6.4 Out-of-distribution detection

Instrument the input distribution independently of the model’s confidence output. Mahalanobis distance in feature space, energy-based scores, and population stability index or Kolmogorov-Smirnov statistics on input features provide a signal that is not derived from the same computation that produces the prediction. This addresses the silent failure mode directly.

6.5 Behavioural assurance

Where structural verification is unavailable, strengthen behavioural evidence: metamorphic testing against invariants that must hold regardless of the model’s internals, slice-based evaluation with per-slice performance floors, adversarial and red-team evaluation with documented coverage, and shadow deployment against the incumbent process with measured disagreement rates before cutover.

6.6 Interpretability tooling, with calibrated expectations

Mechanistic interpretability has advanced materially. Sparse autoencoders decompose polysemantic activations into higher-dimensional sparse feature dictionaries, and attribution graphs trace causal paths between those features. MIT Technology Review listed the field among its breakthrough technologies for 2026, and DeepMind’s Gemma Scope 2 extended sparse autoencoder analysis to models of 27 billion parameters.

The limitations are equally documented and should be stated plainly in any assurance case that relies on these methods. Anthropic’s circuit tracing work on Claude 3.5 Haiku produced satisfying explanations for roughly a quarter of the prompts examined. DeepMind’s interpretability team published negative results on downstream sparse autoencoder applications in March 2025 and deprioritised that line of work. Sparse autoencoders trained on identical data learn different feature sets across runs, a substantial obstacle to using them as audit evidence, and dead-feature rates exceeding 70 per cent have been reported on some models. The January 2025 open problems paper, assembled by 29 researchers across 18 organisations, records that a number of interpretability queries are formally intractable. These tools are useful for investigation and for model debugging. They do not currently constitute a compliance artefact.

6.7 Counterfactual explanation for individual decisions

Where a decision affects a person, a counterfactual explanation states the minimal change in inputs that would have produced a different outcome. It is actionable for the recipient, does not require disclosure of model internals, and maps directly onto the contestability requirement in Article 22C(2) and the reasoning in Dun & Bradstreet. Verify that the counterfactual is feasible, that it does not require changing immutable attributes, and that it is stable under small perturbations of the input.

6.8 Documentation, provenance, and decision-level lineage

Publishing a description of a tool is a lower standard than being able to reconstruct a specific past decision. The distinction matters when a decision is challenged years after it was made. Retain, per decision: input payload, model identifier and weight hash, prompt or feature vector version, sampling parameters, output, confidence or conformal set, any human intervention, and the identity of the intervening officer. Model cards and datasheets for datasets cover the artefact level; they do not cover the decision level.

In the United Kingdom, the Algorithmic Transparency Recording Standard is mandatory for central government departments and for arm’s-length bodies delivering public or frontline services, under the scope and exemptions policy published in December 2024, with more than 120 records now published. It should be treated as a disclosure floor rather than as an audit trail.

Free tool

AI Trust, Risk & Governance Dashboard

Bias and fairness, data lineage, LLM security, and access-control posture against industry-standard thresholds, the four domains a decision-level lineage record has to actually cover.

→

6.9 Governance and procurement

ISO/IEC 42001:2023 provides a certifiable management system for AI; ISO/IEC 23894 addresses risk management and the NIST AI Risk Management Framework provides the govern, map, measure, and manage structure. These establish process discipline. They do not by themselves produce evidence about a specific model.

Procurement terms should specify: notification and re-acceptance rights on model update, per-slice performance floors with defined remedies, retention and disclosure of evaluation artefacts, right of independent audit, and allocation of liability for erroneous outputs. Where a hosted general-purpose model is used, pin to a versioned endpoint and treat provider-initiated updates as change events subject to regression testing.

7. Proportionality

Transparency requirements should scale with consequence, not with model complexity.

Consequence class	Example	Minimum position
Reversible, low individual impact	Document classification for routing, internal search ranking	Behavioural monitoring, drift detection, sampled quality review
Material individual effect	Eligibility triage, priority ordering for inspection or service	Interpretable model preferred; counterfactual explanation; recorded human intervention; per-slice fairness testing
Legal or similarly significant effect	Benefit determination, enforcement referral, credit decision	Interpretable model with justification if not; Article 22C safeguards evidenced; full decision-level lineage
Physical safety or system integrity	Protection settings, dispatch, autonomous control	Opaque component advisory only; deterministic verified envelope on actuation; independent OOD monitoring; documented reversion mode

8. Residual limits

Full mechanistic explanation of frontier-scale models is not available and should not be assumed as a near-term dependency in any programme plan. The appropriate engineering posture is to design systems whose assurance argument does not require it. That means placing opaque components where their failure is bounded, detected, and recoverable, and reserving unbounded reliance for components that can be verified by conventional means.

The black box problem is therefore not principally a research problem for the deploying organisation. It is a systems architecture problem, and the decisions that determine exposure are made at the point where the model’s authority within the system is defined.

An opaque model is not disqualified from a system. It is disqualified from being the only thing standing between a decision and its consequence.

---