MemoryWalker: Stop Training Agents on Contexts They Never Saw
Zinco J and colleagues at Alibaba point out that production harnesses like Claude Code and Qwen-Agent compress context mid-rollout, which turns the training object into a tree rather than a sequence, and give both exact and cheap corrections for the resulting train-inference mismatch.
Ask this paper
The conditioning bug, stated precisely. Every eviction branches the effective history. Retaining the rightmost path leaks information from the future; replaying a depth-first traversal trains on contexts the deployed agent never sees.
Two exact corrections. LogitTree uses a segmented K-forward traversal at the cost of K+1 backward passes; a packed 4D attention mask does it in one pass but needs a custom kernel and white-box eviction records.
SDCC is the deployable one. Self-Distillation for Conditioning Consistency minimizes forward KL between the compressed student and a stop-gradient teacher on the reconstructed pre-eviction prefix in a single backward pass, and works against black-box harnesses.
A bound, not just a heuristic. A residual per-junction KL of epsilon gives an O(sqrt(epsilon)) bound on the train-deployment total-variation gap.
Measured across real harnesses. On seven web-search benchmarks with TC-RAG, AgentFold, MemexRL, Claude Code, and OpenCode, naive training inflates the train-rollout log-probability gap on eviction-heavy batches; the exact methods hold the no-compression floor and SDCC closes most of it with higher rollout rewards.
Abstract
Production agent harnesses such as Claude Code and Qwen-Agent compress context during rollout, but training under compression creates a conditioning problem: every eviction branches the effective history, so the learning object is a tree rather than a sequence. Existing linearizations either retain the rightmost path, causing time-travel leakage, or replay a depth-first traversal, causing train-inference mismatch. We introduce two exact, gradient-equivalent corrections: LogitTree, a segmented K-forward traversal, and a packed 4D attention mask. LogitTree requires K+1 backward passes; the 4D mask requires a custom kernel and white-box eviction records. We also propose SDCC (Self-Distillation for Conditioning Consistency), a single-backward-pass variational relaxation. At each eviction, it minimizes forward KL between the compressed student and a stop-gradient teacher on the reconstructed pre-eviction prefix. A residual per-junction KL of epsilon_KL gives an O(sqrt(epsilon_KL)) bound on the train-deployment total-variation gap. SDCC also applies to black-box harnesses. On seven web-search benchmarks with TC-RAG, AgentFold, MemexRL, Claude Code, and OpenCode, naive training inflates the train-rollout log-probability gap, especially on eviction-heavy batches. The exact methods stay at the no-compression floor, and SDCC substantially closes the gap, with lower logit drift and higher rollout rewards.