A Fragility Spectrum for Recursive Language-Model Training
Yangze Liu and Zhongyi Han (Shandong University) show that how quickly a model collapses under recursive training on generated text is a stable property of the checkpoint, and that a short self-iteration test predicts it.
Ask this paper
Setup: 13 public checkpoints share a common corpus for five generations under one fixed recursive contamination protocol.
Five-fold spread: Unique 4-gram ratio after five generations ranges from 0.187 to 0.940 across checkpoints. The ordering holds with Spearman 0.91 to 0.97 across pool compositions and 0.93 to 0.98 across seeds.
No static predictor: Parameter scale does not explain fragility, since a three-size ladder within one family is not monotonic, and none of the static indicators tested predicts it.
Cheap test: Letting a model iterate on its own output for two or three generations predicts its fragility in the larger ecosystem.
Intervention: Tightening top-p nearly stops collapse within three generations across six checkpoints, while data-side filtering only slows it.
Abstract
Model-generated text is finding its way back into training corpora, and there is plenty of evidence that training on such data over and over collapses output diversity. Prior work has studied the phenomenon itself: which protocols and which data mixtures cause collapse. But different models behave very differently under the same process. We fix one recursive contamination protocol and let 13 publicly released checkpoints form an ecosystem that shares a common corpus for five generations. The unique 4-gram outcome after five generations ranges from 0.187 to 0.940 across checkpoints, a roughly five-fold spread: some models are barely touched, others degenerate into repetitive fragments. Changing the composition of the shared pool or mixing in human text keeps the Spearman correlation of the ordering at 0.91--0.97, and changing the random seed keeps it at 0.93--0.98. Whether a model collapses easily under recursive training is, then, a property of the checkpoint itself, and one that has gone largely unexamined. Parameter scale alone does not explain it, since a three-size ladder within one family is not monotonic in size, and none of the static indicators we tested predicts it either. What does work is cheap: let a model iterate on its own output for two or three generations, and its fragility in the larger ecosystem can be inferred from that alone. Collapse speed also responds to intervention. Tightening top-p, which cuts the low-probability tail at generation time, nearly stops collapse within three generations and stabilizes six checkpoints spanning the whole spectrum together, while data-side filtering slows collapse without stopping it.