Free Pause Tokens
John Langford and colleagues (Microsoft Research, Cornell, CMU) give a language model extra compute per next-token prediction by running it in a parallel prediction stream over a weight-shared backbone instead of spending a sequence position on it.
Ask this paper
The trick is where the compute rides: a pause token normally occupies a position. A free pause token rides an existing position, so it costs no context length, no KV cache and essentially no latency.
Measured gain: 2 to 3 centinats improvement in next-token prediction on a 1B parameter model.
Inference flops grow but do not bind: the authors argue the added flops are typically irrelevant because flops are not the active throughput bottleneck, which is the whole argument for calling it free.
The cost is confined to training: additional training compute against an optimized pretraining pipeline is reduced to as low as 1.14x while preserving most of the benefit.
Isoflop, isoparameter and isotoken: the comparison is controlled on all three axes against standard next-token-trained transformers, which is what makes the claim credible.
Abstract
A free pause token gives a language model extra compute to form each next-token prediction (as a pause, or thinking, token does) but carries that compute in a parallel prediction stream over a weight-shared backbone rather than as an extra token in the sequence. It improves next-token prediction by 2-3 centinats in practice on a 1B parameter model. Because the pause rides an existing position instead of adding one, it is free to use: at inference it adds no context length, no KV cache, and essentially no latency with the growth in inference flops typically irrelevant as it is not the active bottleneck on throughput. The only primary cost is in training, where additional training compute versus an optimized pretraining pipeline is reduced to as low as x1.14 while preserving most of the benefits. The result is an isoflop, isoparameter, and isotoken improvement over standard next token trained transformers.