Is Chain-of-Thought Reasoning a Mirage?

Key hypothesis & framework – CoT’s apparent reasoning ability stems from structured inductive biases learned from training data, not inherent reasoning. Its effectiveness is bound by the distributional discrepancy between training and test data. The authors examine three dimensions: task, length, and format generalization, using their controlled DataAlchemy environment to train and test LLMs from scratch under varied shifts.

Findings on task generalization – Performance collapses when faced with novel transformations or element combinations, even under mild shifts. Correct intermediate steps often yield wrong final answers, exposing unfaithful reasoning. Minimal supervised fine-tuning on unseen data can “patch” performance, but this reflects expanded in-distribution coverage, not true generalization.

Findings on length generalization – CoT fails when reasoning chains or text lengths differ from training data, often padding or truncating to match seen lengths. Grouped training data improves robustness more than simple padding, but degradation follows a predictable Gaussian pattern as length divergence grows.

Findings on format generalization – CoT is highly sensitive to prompt variations. Token insertions, deletions, or modifications, especially in elements and transformation tokens, cause steep performance drops, indicating reliance on surface form.

Temperature & model size – Results hold across model scales and temperature settings; distributional limits remain the bottleneck.

Implications – CoT is a brittle, distribution-bound pattern matcher producing “fluent nonsense” under shifts. Practitioners should avoid over-reliance, use rigorous OOD testing, and recognize fine-tuning as a temporary patch rather than a path to robust reasoning.