Why Language Models Hallucinate
Free while signed in. Answers cite the passages they came from.
The paper argues that hallucinations are not mysterious glitches but the predictable result of how LLMs are trained and evaluated. Pretraining creates statistical pressure to make errors, and post-training benchmarks often reward confident guessing over honest uncertainty. The fix is to realign mainstream evaluations to stop penalizing abstentions.
Pretraining inevitably produces some errors. The authors reduce generation to a binary “Is-It-Valid” classification problem and show a lower bound: the generative error rate scales with the misclassification rate in that classifier. Even with error-free corpora, optimizing cross-entropy yields calibrated base models that still generate errors rather than always saying “I don’t know.”
Arbitrary facts drive a floor on hallucinations. For facts with no learnable pattern (for example, specific birthdays), the paper links hallucination rates to the “singleton rate” in training data. If many facts appear only once, a calibrated base model will hallucinate on at least that fraction of such prompts. This generalizes Good-Turing style missing-mass reasoning and recovers prior results while adding prompts and IDK.
Model class limitations also matter. When the model family cannot represent the needed distinctions, errors persist. The paper formalizes this via an agnostic-learning bound and gives simple cases like multiple choice, where even optimal thresholding leaves a fixed error tied to model capacity, with an example showing classic n-gram models must fail on certain context dependencies.
Post-training often reinforces guessing. Most popular benchmarks grade in a binary correct-incorrect fashion and give zero credit to abstentions, so a model that always guesses can outperform one that withholds uncertain answers. The authors survey widely used leaderboards and find that abstentions are largely penalized, explaining why overconfident hallucinations persist despite mitigation efforts.
Proposed fix: explicit confidence targets. Incorporate clear penalties for wrong answers and neutral credit for IDK directly into mainstream evaluations, instructing models to answer only above a stated confidence threshold. This promotes behavioral calibration, where models choose between answering and abstaining according to the target confidence, and should steer the field toward more trustworthy systems.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack