Prefix Sliding for efficient test-time scaling
Niklas Muennighoff and an eighteen-author team show that most intermediate reasoning tokens stop mattering as reasoning continues, and cap memory by keeping only the prefix and a sliding window of recent tokens.
Ask this paper
The empirical claim comes first: Most intermediate reasoning tokens lose importance as the model reasons further. Prefix Sliding is the consequence of that measurement rather than an architectural bet made in advance.
Keep instructions, keep recent thought, drop the middle: The prefix holds the task instructions and available tools; the trailing window holds what the model is currently working on. Everything between is discarded during generation.
Constant memory regardless of thinking length: Total memory is capped no matter how long the model reasons, which is what makes long-horizon test-time scaling affordable instead of quadratically punishing.
Why it matters: Tools live in the prefix, which is exactly why this maps onto agent loops. An agent that can reason indefinitely at bounded KV cost changes the economics of long-horizon tasks.
Abstract
Test-time scaling uses extra test-time compute to improve performance, such as letting language models reason longer when solving a problem. As models keep the entire reasoning trace in memory via full attention, hard tasks that need long thinking can be prohibitively expensive. However, we find most intermediate reasoning tokens lose importance as the model continues reasoning. This calls into question whether retaining them is worth the cost. Based on this insight, we propose Prefix Sliding, which discards tokens during reasoning that are not part of the prefix or the window of the last few thousand tokens. The prefix has key instructions and tools available to the model, while the most recent tokens are the current reasoning the model is working on. This caps the total memory requirement regardless of how long the model reasons, allowing for efficient long-horizon test-time scaling. Without training, Prefix Sliding can make existing models 3x faster while maintaining performance. Training with Prefix Sliding using reinforcement learning can achieve better performance by enabling scaling to reasoning traces beyond a hundred thousand tokens. Ablations show Prefix Sliding outperforms summarizing intermediate tokens or vanilla sliding window. Our code is at https://github.com/Muennighoff/prefix-sliding