One-sentence Summary

Researchers from Tsinghua University, Alibaba, and Northeastern University propose RAM, a context compression framework that adopts a human-inspired hybrid reading strategy by partitioning contexts into segments and encoding them with the input query in parallel, fully retaining high-relevance segments (close reading) while compressing low-relevance ones into query-guided summary vectors (skimming), and refine the close-reading/skimming decision boundary via a contrastive learning objective, achieving superior performance on question answering and summarization benchmarks across two backbones with up to a $12\times$12× end-to-end speedup on long inputs (average length $16\,\text{K}$16K; maximum length $32\,\text{K}$32K).

Key Contributions

• RAM is a context compression framework that combines explicit textual segments and implicit summary vectors by partitioning the context and encoding each segment in parallel with the input query, fully retaining high-relevance segments while compressing low-relevance ones through query-guided summarization.

• A contrastive learning objective models query-segment relevance through positive and negative pairs, enabling a more robust decision boundary between close reading and skimming and improving compression quality.

• RAM outperforms existing baselines on multiple question answering and summarization benchmarks across two backbones, delivering up to 12x end-to-end speedup on long inputs (average length 16K; maximum 32K) compared to the lightweight baseline Provence.

Introduction

Large Language Models (LLMs) have transformed natural language processing, but the growing use of prompt engineering techniques like Retrieval-Augmented Generation (RAG) means input prompts now routinely stretch to tens of thousands of tokens. Deploying LLMs in these long-context scenarios introduces two core problems: the quadratic time complexity of self-attention in Transformer architectures makes computation increasingly inefficient, and the inherent redundancy of natural language, amplified at long lengths, degrades model performance.

Context compression has emerged as a promising remedy by shrinking sequence length and filtering out irrelevant content. Existing approaches fall into two camps. Task-agnostic methods compress without any query guidance, which risks discarding query-relevant information. Task-aware methods use the query to guide compression and preserve key information, but they face their own obstacles: most either process the entire long context in one pass or rely on autoregressive compression, both of which are computationally expensive, and they struggle with a trade-off between retaining key details and keeping the output interpretable. Some methods prune low-relevance tokens outright, which sacrifices information, while others compress context into implicit semantic vectors, losing the natural language format that users can inspect.

The authors address these limitations by drawing on cognitive science and modeling human reading behavior, where people close read content relevant to their current goal and skim background information. They propose RAM (Read As HuMan), which formalizes context compression as a hybrid reading strategy. RAM partitions a long context into segments and processes all segments in parallel with the input query, avoiding the efficiency bottlenecks of full-sequence loading or autoregressive compression. It then adaptively treats each segment based on query relevance: highly relevant segments are fully retained in natural language to preserve key information, while less relevant segments are compressed into compact, query-guided implicit summary vectors that capture only the semantic gist. These explicit and implicit representations are concatenated into a hybrid contextual representation for the decoder. To sharpen the distinction between the two reading modes, the authors introduce a contrastive learning objective over annotated positive and negative query-segment pairs. RAM achieves superior performance across multiple QA and summarization benchmarks under varying compression budgets on two backbones, and delivers roughly 12x end-to-end speedup on long inputs (average length 16K, maximum 32K) compared to a lightweight baseline.

Method

The authors propose RAM, a novel context compression framework that mimics human reading behavior: it closely reads highly relevant segments while skimming over less relevant ones via query-guided compression. RAM operates within an encoder–decoder architecture and consists of two core stages: query-aware parallel encoding of context segments, and adaptive compression and training. This design avoids the quadratic cost of full-sequence attention and eliminates the need for iterative compression.

3.1 Query-Aware Parallel Encoding

Given a query $Q$Q and a long context $C$C segmented into $N$N equal-length chunks $\{ S_1, S_2, \ldots, S_N \}${S1​,S2​,…,SN​}, RAM processes all segments in parallel with the query using a shared encoder. The original input sequence $\{ S_1, S_2, \ldots, S_N, Q \}${S1​,S2​,…,SN​,Q} is passed through a trainable LLM-based encoder (e.g., LLaMA or Qwen) to obtain contextualized hidden states. Let $\mathbf{H}$H denote the last-layer hidden states.

For each text sequence, a compact representation is constructed by mean-pooling its last hidden states. For segment $S_i$Si​ with length $L_{\mathrm{seg}}$Lseg​, its representation is:

where $\mathbf{h}_{i,j}$hi,j​ is the last hidden state of the $j$j-th token in $S_i$Si​. The query representation $\mathbf{r}_q$rq​ is obtained similarly by mean-pooling the query token hidden states. This parallel encoding step ensures that the query is contextualized against each segment independently, providing a relevance signal without full-sequence attention.

3.2 Adaptive Compression and Training

Based on relevance to the query, RAM dynamically decides whether to preserve each segment verbatim (“close reading”) or compress it into a single vector (“skimming”), guided by a learnable selection mechanism.

Query-Guided Attention

The relevance of each segment to the query is computed via cosine similarity between $\mathbf{r}_q$rq​ and each $\mathbf{r}_i$ri​, followed by a softmax to obtain segment probabilities:

where $\tau$τ is a temperature parameter. Given a sampled compression rate $\alpha$α (sampled from $\{2, 4, 8, 16, 32\}${2,4,8,16,32}), the number of segments $k$k to fully preserve is determined by:

where $L_{\mathrm{org}}$Lorg​ is the token length of the original input. The top-$k$k segments with the highest $p_i$pi​ are selected for verbatim retention; the remaining segments are compressed.

Query-Guided Compression (Skimming)

For segments marked for skimming, a query-aware weighted average of token hidden states is computed:

where $\cos(\cdot, \cdot)$cos(⋅,⋅) denotes cosine similarity. This yields a single embedding $\mathbf{c}_i \in \mathbb{R}^d$ci​∈Rd per skimmed segment, where $d$d is the hidden dimension.

Hybrid Compressed Representation Construction

Let $\mathcal{K} \subseteq \{1, \ldots, N\}$K⊆{1,…,N} denote the set of segment indices selected for close reading, and let $\bar{\mathcal{K}}$Kˉ be its complement. For each segment $i$i, the contribution to the compressed context representation is:

where $\mathbf{e}(\cdot)$e(⋅) denotes the word embedding lookup in the decoder LLM, $\mathbf{c}_i$ci​ is the query-guided compressed vector, and $\mathbf{W}_{\mathrm{align}} \in \mathbb{R}^{d \times d}$Walign​∈Rd×d is a trainable semantic alignment matrix applied only to skimmed segments to bridge the representation gap between explicit tokens and implicit vectors. The indicator $\mathbf{I}_{\{\cdot\}}$I{⋅}​ selects the appropriate representation path per segment. The final compressed context is formed by concatenating all segment representations in their original order:

where $L_c = \sum_{i \in \mathcal{K}} L_{\mathrm{seg}} + |\bar{\mathcal{K}}|$Lc​=∑i∈K​Lseg​+∣Kˉ∣ is the total length after compression. This hybrid representation preserves verbatim tokens for high-relevance segments while replacing low-relevance ones with compact, query-guided summaries, achieving both fidelity and efficiency.

Training Objective

The compressed memory $\mathbf{M}$M is concatenated with the query embeddings and answer tokens, then fed into the decoder for standard teacher-forced language modeling. Let $\mathbf{Y} = [y_1, \ldots, y_{N_a}]$Y=[y1​,…,yNa​​] denote the answer token sequence. The language modeling loss is:

where $P(y_t \mid \cdot)$P(yt​∣⋅) is derived from the softmax-normalized logits, and the loss is computed only over answer positions.

To improve relevance discrimination, a contrastive loss is introduced during training. Given ground-truth positive/negative segment labels (e.g., from answer span annotations), the loss encourages higher similarity between the query and positive segments, and lower similarity with negative ones:

where $\mathcal{P}$P is the set of positive segment indices and $\tau$τ is the same temperature parameter used in query-guided attention. The total training objective combines both terms:

This design enables RAM to achieve high compression efficiency, strong task performance, and natural language format interpretability.

Efficiency Analysis

By segmenting the long context and encoding each segment with the query in parallel, followed by “skimming” low-relevance segments via lightweight compression, RAM fundamentally avoids the bottlenecks of conventional approaches that either process the full input in one shot or rely on autoregressive iteration. The entire pipeline consists of two stages: (1) compression, which includes query-aware parallel encoding and adaptive compression, and (2) decoding, whose floating-point operations (FLOPs) can be modeled separately.

Compression Stage

This stage involves two core components. First, the original context $C$C of length $L_{\mathrm{org}}$Lorg​ is split into $N$N segments, each encoded in parallel alongside the query $Q$Q (length $L_q$Lq​). Second, a query-guided attention mechanism computes a relevance score for each segment to decide whether to retain it as-is or compress it into a compact vector. Compressed segments are aggregated via a lightweight operation (e.g., weighted average) into a single summary vector. Let $L_c$Lc​ denote the total effective length fed to the decoder (i.e., sum of retained token lengths and number of skimmed vectors), and $\alpha = L_c / L_{\mathrm{org}}$α=Lc​/Lorg​ the compression ratio. The FLOPs of this stage are:

where $F_{\mathrm{ParaEnc}}(Q, \mathbf{C})$FParaEnc​(Q,C) denotes the cost of parallel encoding. Since each segment has length $L_{\mathrm{seg}}$Lseg​ and is processed independently, its complexity is $O(N \cdot L_{\mathrm{seg}}^2) = O(L_{\mathrm{seg}} \cdot L_{\mathrm{org}})$O(N⋅Lseg2​)=O(Lseg​⋅Lorg​), which is substantially lower than the $O(L_{\mathrm{org}}^2)$O(Lorg2​) cost of full-sequence encoding (note $N \cdot L_{\mathrm{seg}} = L_{\mathrm{org}}$N⋅Lseg​=Lorg​ and $L_{\mathrm{seg}} \ll L_{\mathrm{org}}$Lseg​≪Lorg​). Meanwhile, $F_{\mathrm{QueryAttn}}(Q, \mathbf{R})$FQueryAttn​(Q,R) (the cost of query-guided attention over $N$N relevance scores) incurs only $O(N)$O(N) computational overhead, which is linear in the number of segments.

Decoding Stage

Assuming the generated answer has length $L_a$La​, decoding requires $L_a$La​ forward passes. The FLOPs of the $i$i-th pass depend on the compressed context length $L_c$Lc​ and query length $L_q$Lq​:

where $\mathbf{M}$M denotes the mixed input (retained tokens and skimmed vectors). The total FLOPs are therefore:

Empirical results show that under the same compression ratio, RAM achieves significantly lower end-to-end latency compared to task-relevant baselines.

Experiment

The experiments evaluate RAM against task-specific and task-agnostic compression baselines on question answering and summarization benchmarks, using LLaMA-3.1-8B and Qwen3-4B backbones. RAM consistently outperforms all baselines under various compression ratios (2x to 32x) and across different model architectures, while also maintaining strong performance on inputs exceeding its training length, demonstrating robust length extrapolation. Efficiency analysis shows RAM reduces inference costs via parallel encoding and query-guided adaptive compression, and ablation studies confirm that each component, including skimming, close reading, query-aware compression, and contrastive learning, is essential to its effectiveness. Overall, RAM proves to be both accurate and efficient, with stable performance across diverse compression settings where baseline methods degrade.

RAM consistently outperforms existing compression baselines across multiple QA benchmarks and a summarization task under different compression constraints, achieving state-of-the-art exact match and F1 scores in most settings. It also maintains stable performance across different backbone models, indicating robustness to model choice. RAM surpasses compression baselines on all benchmarks under both 4x and 8x constraints, with best exact match and F1 in most cases. RAM's performance is stable across LLaMA-3.1-8B and Qwen3-4B backbones, showing robustness. Even when slightly below the fine-tuned original prompt, RAM still beats a task-related baseline after fine-tuning.

On four QA benchmarks with a Qwen3-4B backbone, RAM consistently outperforms a fine-tuned task-specific baseline under both 4x and 8x compression constraints, and often nears or exceeds the fine-tuned original prompt. Its advantage over the baseline grows at higher compression, indicating strong semantic preservation and robustness. RAM surpasses the fine-tuned Provence decoder on all benchmarks under both 4x and 8x compression. Even when RAM trails the fine-tuned original prompt, it remains competitive, and on NarrativeQA it exceeds it at both compression levels. Under 8x compression, RAM's scores drop less than the baseline's, showing better resilience to aggressive compression.

RAM achieves exceptionally low and stable compression latency across all compression ratios, drastically outperforming baseline methods. Its compression latency remains near 0.1 seconds regardless of the compression constraint, while baselines like EXIT incur hundreds of seconds and LongLLMLingua scales down but stays much higher. This efficiency stems from parallel segment encoding and query-guided adaptive compression. RAM's compression latency is around 0.1 seconds across all compression constraints, orders of magnitude lower than all baselines. EXIT shows consistently high compression latency near 300 seconds, while Provence stays around 2.1 seconds and LongLLMLingua decreases from 24.3 to 4.7 seconds as compression becomes more aggressive. RAM's compression latency is nearly invariant to the compression ratio, indicating robust efficiency even at high compression levels.

An ablation study on two datasets shows that removing any core component degrades performance, with the close reading mechanism causing the largest drop. Query-guided skimming outperforms average pooling, and contrastive learning is essential for distinguishing segments that require different processing. Removing close reading leads to the sharpest performance decline, especially on NaturalQA where EM drops from 62.41 to 46.40. Replacing query-guided skimming with average pooling reduces scores, confirming that query-aware compression retains more relevant information. Omitting contrastive learning also markedly hurts results, indicating its role in preventing overcompression of important content.

RAM consistently outperforms existing compression baselines on QA and summarization benchmarks under 4x and 8x compression, achieving state-of-the-art exact match and F1 scores in most settings while maintaining stable performance across different backbone models. Its compression latency remains near 0.1 seconds regardless of compression ratio, orders of magnitude faster than baselines, due to parallel encoding and query-guided adaptive compression. Ablation studies show every core component is essential, with close reading being most critical, query-guided skimming outperforming average pooling, and contrastive learning preventing overcompression of important content.