Against All Likelihood
David Lopez-Paz at FAIR, Meta argues that the truth of a statement should be judged by its invariance across relevant changes of the environment, and connects this to out-of-distribution generalization and continual learning in language models.
Ask this paper
Argument. Maximum likelihood is neither necessary nor sufficient for truth. Unlikely events happen every day, and frequently repeated statements can be false.
Philosophy. The essay draws on Nozick on objective facts, Wigner on invariance and the structural realism of Poincaré and Cassirer, which tie objectivity to invariance.
Machine learning link. It connects this view to invariant learning and out-of-distribution generalization, and notes that loss and accuracy can move in opposite directions.
Proposal. It sketches how invariance could be tested at inference time with language models and discusses implications for continual learning.
Abstract
Maximum likelihood is the dominant training principle for large language models, yet likelihood is neither necessary nor sufficient for truth. Unlikely events happen every day, and frequently repeated statements can be false. Drawing on a philosophical tradition that ties objectivity to invariance rather than probability (from Nozick's account of objective facts and Wigner's principles of invariance to the structural realism of Poincaré and Cassirer), we argue that the truth of a statement should be judged not by its probability under a single distribution, but by its invariance across relevant transformations of the environment. We connect this view to machine learning literature on out-of-distribution generalization and invariant learning, sketch how invariance could be tested at inference time with language models, and discuss implications for continual learning.