Trace as State: Reasoning Traces as Conditional States for Long-Context Transformers
Xu Zou (Z.ai) and Jie Tang (Tsinghua) formalize why causal transformers struggle when the task state is only discovered late, and show that simply moving a reasoning trace in front of the long context is worth enormous accuracy.
Ask this paper
A memory separation result: For causal state update processors, providing the condition first can require exponentially less memory in the worst case than providing it last. The position of the condition is not a prompt-style preference, it is a complexity property.
Trace as State: Collected reasoning traces act as a textual proxy for task state and are placed before the long-context block on a fresh pass, letting previously derived information guide rereading.
A properly matched control: Trace Append uses the identical trace after the context. Trace as State wins in 26 of 27 model-task-metric combinations across three models and three long-context datasets.
The GraphWalks numbers are dramatic: DeepSeek V4 Pro Preview goes from 29.2 percent initial and 43.0 percent with Trace Append to 81.8 percent with Trace as State; GLM-5.2 goes from 66.4 and 83.2 to 100.0.
Why it matters: Cheap, architecture-free, and directly actionable for anyone doing multi-pass long-context work. Put what you learned before the document, not after it.
Abstract
Transformers process information causally, but long-context reasoning may depend on task state discovered only later. We formalize this mismatch through conditional state update tasks. For causal state update processors, providing the condition first can require exponentially less memory in the worst case than providing it last. Motivated by this principle, we introduce Trace as State. We use collected reasoning traces as a textual proxy for task state and place it before the long-context block on a fresh pass, allowing information derived previously to guide rereading. We conduct extensive experiments on Trace as State and Trace Append, a matched control that uses the same task state proxy but put it after the context. Across three models and three long-context datasets, Trace as State outperforms Trace Append in 26 of 27 reported combinations of model, task, and metric. On GraphWalks Parents, exact match lifts DeepSeek V4 Pro Preview from 29.2% on the initial pass and 43.0% with Trace Appendto 81.8% with Trace as State, and from 66.4% and 83.2% to 100.0% for GLM-5.2. These results show that placing traces before the context can improve long-context reasoning while retaining the causal transformer structure.