One-sentence Summary

Sun Yat-sen University and Peng Cheng Laboratory propose MiniLongBench, a compressed long context understanding benchmark created by pruning LongBench with a data compression method tailored for sparse long-text data, reducing evaluation cost to $4.5\%$4.5% of the original while maintaining an average rank correlation coefficient of $0.97$0.97 with LongBench results.

Key Contributions

• The paper introduces MiniLongBench, a compact long-context understanding benchmark distilled from LongBench, containing 237 test samples across six major task categories and 21 distinct tasks.
• The paper proposes a concise data compression method for long-text data with sparse information and validates significant redundancy in LongBench through random sampling experiments to enable low-cost testing.
• Empirical evaluation of more than 60 LLMs shows that MiniLongBench maintains an average rank correlation coefficient of about 0.97 with LongBench while reducing evaluation cost to 4.5% of the original.

Introduction

Long-context understanding is important for LLMs because applications such as book and paper summarization, document question answering, and repository-level code generation require models to process thousands or tens of thousands of tokens. Existing long-context benchmarks such as LongBench improve evaluation quality, but they are expensive to run because each example contains many tokens and the benchmark includes many test samples, leading to high GPU time and memory costs. The authors identify significant redundancy in these benchmark samples and propose a compact alternative, MiniLongBench, which prunes long-text evaluation data while preserving model ranking. Evaluated on more than 60 LLMs, MiniLongBench reduces evaluation cost to about 4.5% of LongBench while maintaining an average ranking correlation near 0.97.

Dataset

• The authors construct MiniLongBench as a compact long-context understanding benchmark from LongBench, a widely used LCU benchmark containing nearly 5000 test samples across six major task categories: single-document question answering, multi-document question answering, summarization, few-shot learning, code completion, and synthetic tasks.

• MiniLongBench retains the same task diversity as LongBench, covering six major categories and 21 distinct tasks. After compression, it contains 237 test samples, representing an approximate 95% reduction in benchmark size while preserving long-context evaluation coverage.

• The compressed subset is generated with a compression ratio of p = 0.95. The resulting benchmark has an average input length of 6193 words for English data and 10344 characters for Chinese data, showing that the retained samples still preserve a similar long-context scale despite aggressive pruning.

• The dataset source is the original LongBench evaluation corpus. The authors first observe through random subsampling experiments that LongBench has high redundancy, because subsets with substantial sample removal can still correlate strongly with the full benchmark ranking results. This motivates a more stable compression procedure.

• The authors use performance records from language models on LongBench to guide sample selection. More than 60 LLMs are considered overall, and 20 selected LLMs are used to construct MiniLongBench. The remaining LLMs are used for validation. The authors manually choose LLMs with generally strong and broad performance because sample representations depend heavily on model performance records.

• During dataset construction, the original long-text samples are first embedded using a text encoder and reduced with principal component analysis to obtain denser low-dimensional representations. The paper sets the representation dimension to d = 10.

• Continuous evaluation metrics from LongBench are converted into binary correctness signals using a threshold search. This allows each sample’s learned representation to reflect whether models succeed or fail on it, which is then used to identify representative samples.

• The authors cluster the learned sample representations with K-Means and retain cluster centers as the final MiniLongBench test samples. No text cropping or content rewriting is described; the compression is primarily sample-level pruning, where selected cluster centers replace the larger original sample pool.

• The constructed MiniLongBench is used as a low-cost evaluation set for estimating long-context model capabilities. The authors evaluate it against the full LongBench rankings to show that the compressed benchmark can preserve evaluation effectiveness while substantially reducing input token count and computational cost.

Method

The authors leverage the performance records of a set of LLMs on a long context understanding benchmark to build a smaller and more stable evaluation subset. The starting observation is that many test samples are redundant, and large random reductions can still preserve ranking correlation, but the resulting estimates have high variance. This motivates a deterministic compression process rather than purely random sampling. Refer to the figure below:

The overall method is organized as a three-stage pipeline. It first densifies sparse long-text inputs, then learns sample representations from LLM performance, and finally clusters and prunes samples to obtain a compact benchmark. Refer to the framework diagram:

In the data preprocessing stage, long-text samples are highly sparse in effective information. To avoid unstable representation learning, the method first embeds each test sample using a long-context text encoder, such as OpenAIEmbedding, and then applies principal component analysis to obtain a lower-dimensional dense initialization. Given the test set $S_L = \{s_j\}_{j=1}^{|S_L|}$SL​={sj​}j=1∣SL​∣​, the initial sample representation is

where $d$d is set to $10$10 in the paper. The figure below summarizes related design choices, including the advantage of a long-context text embedding and the use of a learnable sample bias with zero initialization.

After the text-side representation is initialized, the method performs representation learning through a logistic regression formulation inspired by item response theory. For each LLM $\ell_i$ℓi​ and each test sample $s_j$sj​, the probability that the LLM answers the sample correctly is modeled as

Here, $\theta_i$θi​ is the learnable embedding of LLM $\ell_i$ℓi​, initialized from a standard normal distribution. The parameters $e_j$ej​ and $\beta_j$βj​ are learnable representations of the test sample $s_j$sj​, where $e_j$ej​ starts from the dense embedding produced by PCA and $\beta_j$βj​ is initialized to zero.

When the original evaluation metric is continuous, such as an F1 score or edit distance, the method first normalizes the metric and then converts it into a binary label. It selects a threshold $c$c by solving an approximate optimization problem:

Once $c$c is found, the continuous metric is replaced by

which allows all metrics to be handled within the same binary logistic regression framework.

The parameters $\theta_i$θi​, $e_j$ej​, and $\beta_j$βj​ are updated simultaneously through logistic regression training. The LLM embeddings serve as learnable anchors, while the sample parameters are pushed toward a performance space shaped by how each LLM performs on each sample. This creates a representation that reflects both the content of the long-text sample and the behavioral signal contributed by the participating LLMs.

The choice of participating LLMs matters because the learned sample representations depend directly on the performance matrix. Poorly chosen LLMs can fail to project samples into a meaningful performance space. The figure below summarizes this sensitivity and supports the selection of several generally capable LLMs for constructing the compact benchmark.

After convergence, each test sample is represented by the concatenation of its learned embedding and bias, namely $(e_j, \beta_j)$(ej​,βj​). The method then performs clustering using K-Means with Euclidean distance. The number of clusters is determined by the desired compression ratio $p$p. Specifically, the method keeps

cluster centers, which become the representative samples of the compressed benchmark $S_{\mathrm{mini}}$Smini​. A default compression ratio of $0.95$0.95 is used, meaning that only a small fraction of the original samples is retained.

For evaluating a new LLM $\ell_0$ℓ0​ with the compact benchmark, the method first measures its performance on all samples in $S_{\mathrm{mini}}$Smini​. The same normalization and discretization procedure is applied to these scores. Then a d-dimensional feature vector $\bar{\theta}$θˉ for $\ell_0$ℓ0​ is initialized from a standard normal distribution and fine-tuned using the learned sample parameters $(e_j, \beta_j)$(ej​,βj​) from the mini benchmark. Once $\bar{\theta}$θˉ is adapted to the sample representation space, the performance on the full original benchmark is estimated as

This evaluation step requires only small stored representations and very little computation, because the main work is confined to fine-tuning a low-dimensional LLM vector rather than reevaluating the new model on the full long benchmark.

Experiment

The experiments first evaluate whether long-context understanding benchmarks contain redundancy, using random subsets of LongBench to show that compact samples can often preserve the original model ranking. A compression procedure is then constructed from text embeddings, dimensionality reduction, representation learning over evaluated LLMs, and clustering to select representative test cases. Validation with many LLMs indicates that the compact benchmark largely replicates LongBench rankings while substantially reducing computational cost. Ablations further show that sufficient dimensionality reduction, longer-context embeddings, moderate compression, a learnable sample bias, and an adequate number of training models are important for stable and reliable evaluation.

MiniLongBench is a compact version of LongBench created by pruning test samples while preserving six task categories and 21 tasks. The compressed benchmark has much fewer data points and lower average input length overall, while still keeping long-text inputs in a similar magnitude. This design reduces token counts and computational cost for long-context model evaluation and retains diversity across English and Chinese tasks. MiniLongBench greatly reduces dataset size, with shown task sample counts falling by roughly 95 percent compared with LongBench. Average input lengths are generally smaller but remain within a similar scale, indicating that the compact benchmark still covers long-text scenarios.

The evaluation results show that MiniLongBench reflects relative model strengths across compact long-context tasks, with higher-performing 128k models outperforming smaller or shorter-context models. Model capability gaps are largest on reasoning-heavy tasks such as summarization and synthesis, while code tasks remain relatively easier across models. The results support the claim that MiniLongBench can capture broad evaluation trends at lower computational cost. DeepSeek-V3-128k and GPT-4o-mini-128k consistently lead the task rankings, while smaller or older 8k models lag noticeably. Code-oriented evaluation yields the strongest scores for most models, whereas summarization and synthesis tasks are the most challenging.

MiniLongBench is introduced as a compact evaluation benchmark that preserves the diversity and long-input nature of LongBench while greatly reducing sample size and computational cost. Experiments validate that the benchmark retains the relative performance trends of long-context models, showing stronger results for high-capability 128k models and more pronounced gaps on reasoning-heavy tasks such as summarization and synthesis.