One-sentence Summary

Researchers from Fudan University, The University of Hongampaign, University of Illinois Urbana-Champaign, and Shanghai AI Lab propose L-Eval, a standardized benchmark comprising 20 sub-tasks and 508 long documents spanning 3k–200k tokens for evaluating Long-Context Language Models, and show that Length-instruction-enhanced (LIE) evaluation and LLM judges better correlate with human judgments than existing metrics.

Key Contributions

• L-Eval is introduced as a standardized evaluation suite for long-context language models, comprising 20 sub-tasks, 508 long documents, and more than 2,000 human-labeled query-response pairs across task types, domains, and input lengths from 3k to 200k tokens.

• Length-Instruction-Enhanced evaluation is developed to guide models toward desired answer lengths, and experiments show that this approach and LLM judges correlate with human judgments better than standard n-gram metrics.

• A study of 16 LLMs is conducted on L-Eval, revealing gaps between open-source and commercial models, weaker performance of context-extension methods on open-ended tasks, end-to-end long-context models outperforming retrieval-based systems, and mixed effects of training-free scaled positional embeddings on retrieval and reasoning.

Introduction

Long-context language models are increasingly important because many real-world applications require reasoning over documents, meetings, codebases, and other extended inputs. Existing benchmarks for these models are often too limited, relying on short prompts, narrow task distributions, and automatic scores such as ROUGE, which can fail to capture meaningful differences in model quality.

The authors introduce L-Eval, a standardized benchmark for long-context evaluation. L-Eval covers closed-ended and open-ended tasks, expands task diversity through newly annotated samples, and uses multiple evaluation methods, including reference-based metrics, LLM judges, and human evaluation. The paper also proposes Length-Instruction-Enhanced evaluation to reduce length bias in automatic scoring and studies 16 recent models to reveal gaps in model capability and evaluation fidelity.

Dataset

• The authors construct L-Eval as a 20-task long-context benchmark, grouping the data into three construction paths: newly annotated tasks, re-annotated public datasets, and existing tasks with correction and filtering.

• Dataset composition and sources:

  • Four subsets are annotated from scratch: Coursera, SFiction, CodeU, and LongFQA.
  • Five subsets are re-annotated from public datasets: GSM(16-shot), QuALITY, Openreview, SPACE, and TopicRet.
  • Eleven additional tasks are derived from existing datasets and processed through a manual review pipeline.

• Key details for the annotated subsets:

  • Coursera: Built from Coursera lecture subtitles in big data and machine learning; questions, options, and gold answers are authored by the paper’s authors; it is a closed-ended task with multiple correct options.
  • SFiction: Built from science-fiction contexts; it uses True/False questions where many answers conflict with real-world facts, testing whether models rely on the given context rather than parametric knowledge.
  • CodeU: Built from long Python programs that call NumPy functions and a custom string-processing codebase; function names are replaced with placeholders such as Op1 and Op2 to prevent models from inferring behavior from descriptive names.
  • LongFQA: Built from public earnings call transcripts collected from the Investor Relations pages of six company websites; it targets long-context financial question answering.

• Key details for the re-annotated public datasets:

  • GSM(16-shot): Uses 100 grade-school math problems from GSM8k and constructs 16 chain-of-thought in-context examples, with eight examples from chain-of-thought-hub and eight created by the authors.
  • QuALITY: Existing QA data is augmented with new synthesis instructions to test global context modeling.
  • Openreview: Papers are collected from openreview.net, with tasks such as writing an abstract, summarizing related work, and giving feedback.
  • SPACE: A review summarization dataset is extended with diverse user queries covering different aspects of the input.
  • TopicRet: A synthetic retrieval task is made harder by asking for the second or third topic instead of the first topic, making it more useful for distinguishing model abilities.

• Processing and filtering for existing tasks:

  • The authors use Claude-100k to filter samples from the 11 existing-task sources.
  • Claude is prompted to generate an answer and supporting evidence from the document.
  • If the answer conflicts with the ground truth or cannot be inferred from the context, the sample is manually reviewed, then either corrected or discarded.
  • No cropping strategy or special truncation method is described; the main processing focuses on annotation, re-annotation, synthesis instructions, function anonymization, and error correction.

• Scale, schema, and usage:

  • The dataset covers diverse task types, including multiple-choice QA, True/False QA, few-shot math, code understanding, dialogues, extractive QA, abstractive QA, and summarization.
  • Inputs span domains such as law, finance, academic papers, lectures, conversations, news, Python code, novels, and meetings.
  • Average inputs range from 4k to 60k tokens, with some samples reaching nearly 200k tokens.
  • The section presents L-Eval as a benchmark dataset; it does not describe a training split, mixture ratios, or fine-tuning use, and the data is used to evaluate models on long-document tasks.

Method

The authors build the L-Eval workflow as a data-centric evaluation pipeline rather than a trainable model architecture. The method has two main modules: benchmark construction with task annotation and automated filtering, and controlled automatic evaluation that reduces length bias in open-ended scoring.

The construction pipeline first separates tasks into closed-ended and open-ended formats. Closed-ended tasks are designed for objective scoring, while open-ended tasks target summarization and abstractive question answering. For newly annotated closed-ended tasks, the authors create domain-specific long-context problems. Coursera uses lengthy lecture subtitles and requires models to select multiple correct options from subtle evidence. SFiction tests whether models follow contextual information even when it contradicts pretrained factual knowledge. CodeU evaluates code understanding by asking models to predict outputs of long Python programs with anonymized function names such as Op1, Op2, and OpN, preventing reliance on semantic function labels. LongFQA uses long earnings-call transcripts from financial documents. These task designs ensure that the benchmark covers domains and task forms where reasoning must be grounded in the full input context.

For tasks derived from existing datasets, the authors introduce a filtering and correction workflow. They use a long-context model to generate both an answer and supporting evidence for each sample. If the generated answer strongly disagrees with the ground truth, or if the model indicates that the answer cannot be inferred from the provided document, the sample is sent for manual review. Reviewers then either discard the sample or correct it. This module improves label consistency and removes noisy question-answer pairs that cannot be reliably solved from the long input alone.

After the benchmark is constructed, the evaluation module addresses a key limitation in open-ended scoring: length bias. In preliminary experiments, models tend to produce verbose responses with explanations, causing mismatched granularity between predictions and reference answers. This harms n-gram-style metrics because token overlap can be distorted even when the factual content is useful. The authors therefore introduce Length-Instruction-Enhanced evaluation, or LIE evaluation. The core idea is to expose the desired output length directly in the prompt. If the reference answer contains $w_y$wy​ words, the model receives an instruction such as producing a summary of exactly $w_y$wy​ words. The metric then measures content accuracy rather than accidentally rewarding or penalizing arbitrary generation length.

To validate this evaluation design, the authors compare automatic metrics against human judgments on open-ended questions. Three annotators score outputs from multiple models, and the authors compute the Kendall Tau coefficient $\tau$τ between each automatic metric and the average human score. The results show that length control substantially improves agreement with human evaluation.

As shown in the figure below:

The bar chart illustrates the gain from adding length instructions to several automatic metrics. Metrics that originally fail to align with human scores become more correlated after the prompt includes the target length. The gain is especially pronounced for ROUGE-L and F-1, while GPT-4 evaluator scores also increase to a perfect or near-perfect correlation level.

The authors further examine whether model rankings become more stable under different scoring systems. They compare six evaluation schemes, including human averages, individual human scores, LLM judges, and n-gram metrics. The radar comparison visualizes how different models are ranked across evaluators.

Refer to the radar comparison:

The radar plots show that rankings without length control are more irregular, reflecting weak correlation among metrics. With length instructions, the ranking patterns become more consistent, indicating that controlled generation length makes automatic evaluation more faithful to human judgment.

Experiment

The experiments evaluate L-Eval using exact-match scoring for closed-ended tasks and human, LLM, and lexical metrics for open-ended tasks, including a length-instruction protocol to test metric reliability. They validate that controlling output length strengthens agreement between automatic evaluators and human judgments and helps expose length bias in lexical scores. Benchmarking 16 models shows that commercial systems generally outperform open-source models, with GPT-4 strongest on closed-ended tasks and Claude strongest on open-ended tasks. The results also indicate that longer-context tuning helps closed-ended generation but remains limited for robust open-ended reasoning, especially with longer inputs and retrieval-sensitive settings.

The table describes L-Eval as a long-context evaluation suite with diverse task types, domains, instruction counts, document counts, and input lengths. It combines closed-ended and open-ended tasks to support more objective and flexible evaluation of long-context language models. The varied lengths and newly added tasks improve coverage of realistic long-prompt scenarios. Task types include multiple-choice QA, true or false QA, few-shot learning, retrieval, code understanding, dialogue, extractive QA, abstractive QA, and summarization. Inputs span many domains and lengths, from moderately long samples to very long documents, giving the suite broader realism than short-prompt benchmarks.

Across the four long-context datasets, large differences between generated answer length and reference length can strongly hurt performance. Some models score poorly mainly because their outputs are much longer or shorter than expected, not necessarily because the content is incorrect. Explicit length control helps reduce this bias and makes evaluation more reliable. Length mismatch with reference answers is a major source of poor scores. Guiding models toward target answer length improves evaluation stability.

The closed-ended results show a clear hierarchy among long-context models. GPT-4-32k achieves the strongest overall performance, while the 100k model is competitive but lower on average. Retrieval-based approaches using shorter 4k context lag behind the full 16k counterpart, and a truncated 2k baseline performs poorly. GPT-4-32k establishes the strongest closed-ended benchmark among the compared models. Encoding the full 16k context is more effective than 4k retrieval-based pipelines. Models using scaled positional embeddings and longer fine-tuning improve closed-ended task scores.

On open-ended tasks judged by GPT-4, Claude-1.3-100k shows the strongest performance, followed by GPT-4-32k and full-context Turbo-16k. Turbo-4k-based retrieval methods improve over the plain Turbo-4k baseline but remain behind models that process the full context, suggesting that long-form generation requires more than simple retrieval. The results also indicate that scaled open-source models still lag on challenging open-ended generation despite gains on closed-ended tasks. Claude-1.3-100k has the highest win rate, outperforming GPT-4-32k and the Turbo baselines. Retrieval-augmented Turbo-4k variants surpass plain Turbo-4k but fall short of full-context Turbo-16k. Open-source models using scaled context techniques do not reliably exceed GPT-3.5 or Turbo on open-ended generation.

Experiments evaluated L-Eval, a diverse long-context suite spanning many task types, domains, and input lengths, and found that answer length mismatches can distort scores, making explicit length control important for reliable evaluation. Closed-ended results show GPT-4-32k as the strongest overall, with full-context processing, scaled positional embeddings, and longer fine-tuning outperforming 4k retrieval or truncated baselines. Open-ended results favor Claude-1.3-100k, followed by GPT-4-32k and Turbo-16k, indicating that retrieval helps but full-context models remain stronger for challenging long-form generation.