A model learns patterns from the data it’s trained on, and if that data under-represents certain groups, languages, or scenarios, or reflects historical decisions that were themselves biased, the model reproduces and often amplifies that skew. It’s not usually a deliberate design choice; it’s an inherited one, which is exactly what makes it easy to miss until it surfaces in a decision that affects a real person.
This is a live, practical risk anywhere AI touches hiring, lending, insurance pricing, or resource allocation, not an abstract ethics debate. A model that performs well on average can still perform badly and consistently for a specific subgroup, and “the average is fine” is not a defence once a regulator or a claimant asks about that subgroup specifically.
Catching it requires deliberately testing outputs across different subgroups, not just checking overall accuracy, and treating an uneven error rate as a defect to fix, on the same footing as a factual hallucination, rather than an unavoidable side effect of the technology.