Context Language Models
Rulin Shao, Hamish Ivison, Nathan Lambert, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Pang Wei Koh and colleagues at the University of Washington and Meta Superintelligence Labs (with MIT) introduce Context Language Models (CLMs), which manage their own context by treating it as a file they can edit with Bash, replacing harness-defined compaction, offloading and retrieval rules.
Ask this paper
Context as a read-write file: The model edits its own context with code, writing loops that prune irrelevant results, defining functions it reuses, and tracking subagents. Recursive language models place a large input in an external variable the model can read, but they do not let it edit its live interaction context. Because several agent contexts can coexist as files, the approach extends to multi-agent systems.
Works zero-shot: Built from existing models with no training, CLMs reach 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus than state-of-the-art context-management strategies, 5% higher scores with 59% fewer FLOPs on the 12-hour EdgeBench, and 65% more improvement for the same compute on a 24-hour agent-swarm task across six repositories.
Learnable in context and in weights: Natural-language instructions evolved through a skill-optimization loop raise held-out accuracy by up to 35.9 points on a context-management task. An online RL method improves Qwen3.5-9B on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs.
Why it matters: Edits in the middle of the context break prefix caching, so the authors co-design Suffix Cache Reuse, which cuts server-side compute by 35% against standard SGLang at matched performance. Context management can move from the harness into the model without raising serving cost.
Abstract
We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to make unrestricted updates to this file. This allows the model to learn what is most important to maintain in context, and naturally extends to multi-agent systems where multiple agent contexts coexist as files. Building CLMs zero-shot with existing models outperforms SOTA context management strategies across a variety of tasks: 11.4% higher accuracy with 21.5% fewer FLOPs on BrowseComp-Plus, 5% higher scores with 59% fewer FLOPs on 12-hour EdgeBench, and 65% greater improvement with the same compute on a 24-hour multi-repository agent-swarm task. Moreover, by shifting context management from external harness control to intrinsic model behavior, CLMs naturally enable both in-context and parametric learning of context-management strategies. We show that CLMs can be steered with natural-language instructions evolved through a standard skill-optimization loop, improving held-out accuracy by up to 35.9 points on a context-management task while reducing compute. We also introduce an online reinforcement learning method for CLMs, improving Qwen3.5-9B performance on BrowseComp-Plus by 47.6% while using 12% fewer FLOPs. Finally, we co-design Suffix Cache Reuse for CLM serving, further reducing server-side compute by 35% relative to standard SGLang at matched performance.