One-sentence Summary

Researchers at Tsinghua University and The Chinese University of Hong Kong propose cache engineering, enabling agents to directly control active KV cache through drop and repos without re-prefilling retained context, and co-design the AKV serving engine for stateful cache reuse; their tool response rolling-drop cache policy improves average benchmark scores by $1.7$1.7 points and reduces average cost by $27.7\%$27.7%, achieving up to $1.2\times$1.2× output throughput and a $1.5\times$1.5× speedup in $P_{95}$P95​ request latency compared with conventional threshold-triggered summarization.

Key Contributions

• Introduces cache engineering, a scheme that lets agents directly control their active KV cache through drop and repos operations without modifying the trajectory or re-prefilling retained context; it formalizes these cache operations and the resulting history-dependent cache states, showing why prefix matching alone is insufficient for correct cache reuse.
• Presents AKV, a serving system that exposes a stateless interface compatible with existing serving APIs while internally managing stateful cache reuse, eviction, and restoration through an extended radix tree and segment-batch attention.
• Across three LLMs and four agent benchmarks, a tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7% compared with threshold-triggered summarization; with AKV, it achieves up to 1.2× output throughput and a 1.5× speedup in P95 request latency, and recursive self-improvement further reduces cost by 7.1%.

Introduction

As LLM agents take on longer, tool-heavy tasks, their accumulated KV caches become a major bottleneck: KV cache reads dominate API cost, and attending to too many or positionally stretched states can hurt task accuracy. Prior work mainly edits the text trajectory through summarization or trimming, but modifying earlier messages invalidates later KV states and forces expensive re-prefill, creating a dilemma between re-prefill cost and retaining useful intermediate context. The authors introduce cache engineering, which directly drops selected KV entries from future attention or adjusts their encoded positions while reusing all retained KV states without re-prefill. They co-design AKV, a serving system with a stateless interface, an extended radix tree, and segment-batch attention to make this control practical for real agent workloads. The approach also supports recursive self-improvement, letting agents search, evaluate, and refine cache management policies.

Method

The authors introduce cache engineering to allow agents to directly control their active Key-Value (KV) cache without rewriting the logical token trajectory or triggering expensive re-prefilling. In standard autoregressive LLM inference, next-token prediction relies on the current token and the preceding tokens' KV cache, formulated as $p_{\theta}(x_{t+1} \mid x_t, \mathcal{C}_{1:t-1})$pθ​(xt+1​∣xt​,C1:t−1​). While context engineering modifies the logical token sequence through trimming or summarization, cache engineering directly modifies the active cache while keeping the logical token sequence unchanged.

Refer to the framework diagram:

As illustrated above, full-context inference reuses all historical KV states, whereas context engineering removes text and recomputes the retained history. In contrast, cache engineering drops selected KV entries from future attention while reusing all retained states without recomputation. This approach alleviates the dilemma between re-prefill cost and task accuracy. Empirical evaluations show that directly dropping older tool-response cache can simultaneously improve accuracy and reduce cost compared to trim-based or summary-based context engineering.

As shown in the figure below:

The design of cache selection reflects message roles, recency, and task needs. Analysis of attention patterns reveals that older tokens and tool responses generally receive less attention than recent tokens and assistant messages.

Refer to the attention analysis:

Based on this, the authors employ a tool-response rolling drop policy, retaining cache for reasoning and recent tool responses while dropping older ones. Furthermore, to prevent position encoding indices from reaching the model's limit as the cache is reduced, the system decouples position encoding indices from logical indices, using a separate operation to reposition cached keys with Rotary Position Embeddings (RoPE) rotations.

The authors define three basic cache operations:

1. Append: The standard cache update that processes a new token $x_{t+1}$xt+1​ and appends its KV entry to the active cache: $\mathrm{append}(x_{t+1}): \mathcal{C} \leftarrow [\mathcal{C}; C_{t+1}]$append(xt+1​):C←[C;Ct+1​].
2. Drop: Removes selected KV states from subsequent attention while leaving retained states, their position encodings, and the logical token sequence unchanged: $\mathrm{drop}(P): \mathcal{C} \leftarrow \mathcal{C} \setminus \{C_i: i \in P\}$drop(P):C←C∖{Ci​:i∈P}.
3. Repos: Updates cached keys' position encodings in logical token order and resets the position index for subsequent append operations. For RoPE, this involves rotating keys from their original position index $p$p to a new index $q$q via $K \leftarrow R(q)R(p)^{-1}K$K←R(q)R(p)−1K.

To provide a deterministic description, these operations are organized into a cache operation sequence $o_{1:m} = (o_1, \dots, o_m)$o1:m​=(o1​,…,om​), where $o_j \in \{\text{append}(x), \text{drop}(P), \text{repos}\}$oj​∈{append(x),drop(P),repos}. Cache states are history-dependent; tokens and retained positions alone do not determine next-token probabilities.

Refer to the trajectory diagram:

As shown in the figure, different trajectories retaining the same tokens (B and C) can result in distinct cache states depending on when drops occurred. The extended radix tree in the diagram shares prefixes but branches according to drop timing to distinguish cached states.

The AKV serving system follows a stateless interface and stateful serving principle. Agents specify cache operations via an extended API that includes both the logical token sequence and the cache operation history. The server handles physical cache reuse, eviction, and restoration internally. To support cache engineering, the system addresses challenges in cache state management. Token prefixes alone no longer determine cache states, so ordinary prefix matching is insufficient. The authors extend the radix tree with drop nodes to record logical token indices whose cache is dropped, preserving prefix sharing while distinguishing later states. Eviction is drop-aware; a cache entry becomes evictable only when no ongoing computation or restoration needs it, prioritizing dropped cache over completed request cache. Restoration of missing cache states is achieved through segment-batch attention, where queries are split at drop trigger points to attend causally to their own tokens and active cache from earlier segments in a single prefill pass.

For reposition-aware management, the system reuses cache after repos by copying cached states and rotating keys without recomputing hidden states.

Refer to the repositioning diagram:

As illustrated, when a request requires repositioned cache, the system creates copies with updated position indices and a reposition node in the radix tree. Positional versions are evicted independently when no longer in use. Restoration uses temporary KV versions, rotating keys and copying values at each layer to ensure segments use the correct position encodings from their original computation.

Experiment

The evaluation compares AKV with two context-summarization baselines across three LLMs on long-horizon agentic benchmarks including BrowseComp-Plus, xbench-DeepSearch, GAIA-Text, and SWE-Verified. AKV achieves higher average success rates while reducing inference cost, and it shows better throughput and latency, especially at high concurrency. The gains come from incrementally dropping older tool-response KV cache instead of abrupt summarization. Ablations show that tool-response drop with repos is the strongest policy, while recursive self-improvement and agent-managed cache control can further lower cost or improve accuracy.

AKV achieves the highest average pass rates across the evaluated models, improving over both baselines while lowering average inference cost. For MiniMax-M2.7, AKV records the highest average score and lowest average cost despite mixed benchmark-level score changes; gains are strongest on GAIA-Text, and costs are lower on most individual benchmarks. AKV improves average success rates over Standard and Text Engineering while reducing average inference cost. For MiniMax-M2.7, AKV has the highest average score and lowest average cost among methods, but Text Engineering leads on BCP and Standard leads on SWE-Verified.

At the reported concurrency levels, AKV achieves lower request latency and time to first token than both Standard and Text Engineering, while also delivering higher input and output throughput. The efficiency advantage widens under higher concurrency, where Standard experiences more KV-cache pressure and re-prefill work. At peak-throughput settings, AKV also outperforms Standard on mean and P95 request latency and on input and output throughput. AKV records the lowest mean and P95 request latency and time to first token among the three methods at the reported concurrency levels. At peak-throughput settings, AKV reaches about 1.4x faster mean request latency and about 1.5x faster P95 request latency than Standard. AKV delivers higher input and output throughput than both baselines across the reported concurrency levels. The latency and throughput gaps over Standard grow at high concurrency because KV-cache pressure increases Standard's re-prefill work.

The ablation shows that dropping tool-response cache with repositories enabled provides the strongest accuracy profile, achieving the best BCP pass rate and matching the best SWE score among the compared strategies. Repositories account for a large BCP improvement, from 55 to 62, while tool-response dropping keeps costs below Standard. When repositories are enabled, dropping reasoning cache alone or together with tool-response cache reduces pass rates on both benchmarks relative to dropping tool-response cache alone. Tool-response dropping with repositories enabled yields the best BCP pass rate and ties the best SWE score among the compared strategies. Adding repositories to tool-response dropping raises BCP pass rate from 55 to 62, with cost still below Standard. With repositories enabled, dropping reasoning cache, either alone or with tool-response cache, underperforms tool-response-only dropping on both benchmarks.

Cache-As-Tool achieves the highest pass rate on the 100-task BCP subset for both models compared with the Standard and text engineering methods. For GPT-OSS-120B, this approach also lowers per-task cost relative to Standard, while MiniMax-M2.7 sees a modest cost increase as it executes more turns. Text engineering produces mixed accuracy effects, decreasing GPT-OSS-120B's score but improving MiniMax-M2.7's score. Cache-As-Tool attains the highest accuracy on the 100-task BCP subset for both models. For GPT-OSS-120B, Cache-As-Tool reduces per-task cost relative to Standard, whereas MiniMax-M2.7's per-task cost rises modestly. Text engineering lowers GPT-OSS-120B's pass rate compared with Standard but improves MiniMax-M2.7's pass rate.

The evaluation compares AKV against Standard and Text Engineering on accuracy, inference cost, latency, throughput, and cache ablations across benchmarks such as BCP, SWE-Verified, and GAIA-Text. AKV consistently achieves the highest average pass rates and lowest inference cost, while also delivering lower request latency, lower time to first token, and higher input and output throughput, especially under high concurrency. Ablations show that enabling repositories and dropping tool-response cache yields the strongest accuracy and keeps costs below Standard, whereas dropping reasoning cache reduces pass rates. On the 100-task BCP subset, Cache-As-Tool attains the highest accuracy for both models, with cost effects varying by model.