A model inversion attack works backwards from a model’s outputs to reconstruct information it was trained on, rather than trying to trick the model into misbehaving the way prompt injection does. By systematically querying a model and analysing patterns in its responses, an attacker can reconstruct approximations of training records the model was never supposed to expose, including, in documented cases, recognisable faces from an image classifier or fragments of personal data from a language model trained on private records.
It’s a harder problem to defend against than most AI security risks because the attack uses the model exactly as intended: legitimate queries, no injected payload, no malformed input for a filter to catch. The vulnerability is structural. A model that has memorised specifics of its training data, rather than learning general patterns, is inherently more exposed, which is why differential privacy techniques during training and tight access control around who can query a model at scale are the standard mitigations, not input filtering.
IBM’s 2026 breach research found model inversion among the costliest categories of AI-specific incident, and traced the access almost entirely back to one gap: models and the pipelines around them held to a lower access-control standard than a production database would ever be allowed. The fix isn’t a smarter model. It’s treating query access to a trained model as sensitive as the data it was trained on.