Most attempts to understand a model treat it as a black box and study its behaviour from outside: probe it with inputs, watch the outputs, infer a pattern. Mechanistic interpretability instead opens the box and tries to map what is actually happening inside, using tools like sparse autoencoders to decompose a layer’s tangled, overlapping activations into a larger set of individually meaningful features, and attribution graphs to trace the causal path those features take between an input and an output.
The field has advanced materially and remains, at the current state of the art, an investigative tool rather than a compliance artefact. Anthropic’s own circuit-tracing work on Claude 3.5 Haiku produced a satisfying explanation for roughly a quarter of the prompts examined, using the best published methodology available. Sparse autoencoders trained on identical data can learn different feature sets across separate runs, and a meaningful share of learned features never activate at all, both of which undercut using the output as audit evidence.
Treat it accordingly: useful for debugging a specific model behaviour or hunting for a planted backdoor, not yet sufficient on its own to satisfy a regulator asking why a model made a particular decision.