Cache Engineering for Agents
Tianyu Fu, Yu Wang and colleagues at Tsinghua University propose cache engineering, in which agents edit their own active KV cache with two operations instead of rewriting their context, and co-design the AKV serving engine that makes those edits reusable.
Ask this paper
Operations. drop excludes selected cache entries from future attention, and repos adjusts the position encodings of retained keys. Neither modifies the trajectory text or forces a re-prefill of retained context.
Serving problem. Identical token sequences can correspond to different cache states depending on the sequence of operations applied, so prefix matching is not sufficient for correct cache reuse. AKV exposes a stateless API and manages stateful reuse, eviction and restoration internally.
Results. A rolling drop of tool responses raises average benchmark scores by 1.7 points and cuts average cost by 27.7% compared with threshold-triggered summarization. AKV gives up to 1.2x output throughput and a 1.5x speedup in P95 latency.
Self-improved policies. Recursive self-improvement of cache policies lowers cost a further 7.1% below the expert-designed policy.
Abstract
Large Language Model (LLM) agents can handle increasingly complex tasks, navigating long trajectories of observations and actions. As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost. Trajectories can also exceed the LLM's maximum context length, requiring control over active cache growth. Prior work uses context engineering to reduce cache size by trimming or summarizing history. However, modifying the trajectory prefix invalidates subsequent cached states and triggers costly re-prefilling of retained context. Task accuracy can also suffer when too little context is retained to limit re-prefill costs. We propose cache engineering as a new scheme. It allows agents to directly control their active KV cache without modifying the trajectory or re-prefilling retained context. Specifically, agents can proactively use drop to exclude selected cache from future attention and repos to adjust retained keys' position encodings. We formalize these cache operations and identify history dependence as a serving challenge: identical token sequences can yield different cache states under different cache operation sequences, making prefix matching insufficient for correct cache reuse. Thus, we co-design the AKV serving engine. It provides a stateless interface compatible with existing serving APIs, while internally managing stateful cache reuse, eviction, and restoration. Experiments show that a tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7%, compared with conventional threshold-triggered summarization. With AKV, it achieves up to 1.2x the output throughput and a 1.5x speedup in P95 request latency. Recursive self-improvement of cache policies further reduces cost by 7.1% over expert design. Our code is at https://github.com/thu-nics/AKV.