A language model doesn’t “know” facts the way a database does; it generates the next most statistically plausible sequence of words. Most of the time that produces a correct answer, because correct answers are usually what plausible text looks like. Sometimes it produces a fabricated case citation, a nonexistent product feature, or a made-up statistic, delivered with exactly the same fluent confidence as a true one.
This is the single biggest reason enterprise AI deployments stall after the pilot stage. A hallucination that reaches a customer, a regulator, or a board deck doesn’t fail quietly; it fails in public, with your name on it.
The fix isn’t a better model; it’s system design: grounding answers in retrieved source documents, building in “I don’t know” as an acceptable output, and putting a human check on any answer above a certain stakes threshold before it goes out.