A black box is not the same as a secret. An organisation can hold the full weight file of a model on its own servers and still have no way to state why a given input produced a given output, because the parameters were learned against a loss function rather than written as rules. Jenna Burrell’s widely cited taxonomy separates this from opacity caused by a vendor withholding information or by a reviewer lacking the expertise to read it. Only the learned, high-dimensional kind resists a fix by disclosure, because no party, including the model’s own developer, holds the explanation.
This distinction matters because it changes what kind of assurance is available. A conventional system can be debugged by locating the responsible logic and verified against a written specification. A black box has no specification to check against, only a distribution to test, which turns a deductive safety argument into a statistical one that quietly stops holding the moment the operating conditions shift.
The practical response is not to wait for the box to become transparent. It is to stop asking the model to explain itself and instead build the check outside it: bound what the model is allowed to decide, and instrument the system around it so a failure is caught from outside rather than by asking the black box why it happened.