One-sentence Summary

Bar-Ilan University and Allen Institute for AI introduce a QA reasoning framework that isolates the effect of input length by padding identical samples with varying lengths, types, and locations, revealing that LLM reasoning degrades at inputs far shorter than their technical maximum and that next-word prediction negatively correlates with reasoning accuracy.

Key Contributions

• The paper introduces FLenQA, a QA reasoning dataset that isolates input length by extending identical task samples with controlled padding of varying length, type, and location, while keeping base task difficulty simple.
• The results show that LLM reasoning accuracy can fall sharply before the technical maximum context length, with average accuracy dropping from 0.92 to 0.68 around 3,000 tokens. This degradation persists across padding variants, and Chain-of-Thought prompting raises scores but does not close the long-context gap for most models.
• Next-word prediction performance on long inputs is shown not to track downstream long-context reasoning performance. The analysis also identifies failure modes including instruction-following breakdowns, answer-before-reasoning under Chain-of-Thought prompting, false-answer bias, and reduced use of relevant information as inputs grow.

Introduction

Large language models increasingly advertise long context windows, making it important to verify whether their reasoning remains reliable when the same task is embedded in longer prompts. This matters for applications that require text-based or multi-step reasoning over extended documents. Prior evaluations often changed both the input length and the task difficulty, or used benchmarks with fixed lengths, making it hard to isolate the effect of length alone. Next-token prediction also proved unreliable as a proxy for downstream reasoning performance.

To address this, the authors introduce FLenQA, a question answering dataset that keeps the underlying reasoning task constant while embedding the required information in different amounts of irrelevant context. They leverage this setup to show that reasoning accuracy can degrade sharply well before a model's technical context limit. They also find that chain of thought prompting improves performance on shorter inputs but does not fully close the reasoning gap on longer inputs for most tested models.

Dataset

• The authors use FLenQA, or Flexible Length Question Answering, a benchmark created to study how input length affects reasoning in language models. It contains 100 base instances for each of three reasoning tasks, and every task is balanced over True/False labels.

• The dataset has three sources and tasks. Monotone Relations is a new task inspired by monotonic relation reasoning and is generated with random names from the Faker Python library and handcrafted relation types. People In Rooms is also new, simplified from bAbI-style path reasoning, with random person names and hand-selected mutually exclusive room-property pairs. Simplified Ruletaker adapts the Ruletaker theorem-proving format using an explicit rule, two fact sentences, and a question.

• Each base instance has a controlled prompt schema: an optional prefix, two key paragraphs, and an optional suffix. Each key paragraph is a coherent paragraph built around one key sentence needed for the answer. The parts are joined by newlines before being sent to the model.

• The authors filter and validate the data for reasoning reliability. Each task is built so the answer requires both key paragraphs, while decomposable or single-span solutions are avoided. Key sentences are expanded into paragraphs using GPT-4, prompted to add no new information, and manually verified. Similar-task padding is filtered by excluding paragraphs containing entities from the key paragraphs, and room-property pairs are chosen to be mutually exclusive to avoid ambiguity.

• Length variations expand each base instance to roughly 250, 500, 1000, 2000, and 3000 tokens. The authors add irrelevant padding from three background sources: duplicated copies of the key paragraphs, paragraphs sampled from other instances of the same task, or continuous text from the Books Corpus. The Books Corpus text is cut at sentence boundaries before injecting the key paragraphs.

• Each length variant also varies where the key paragraphs appear: at the beginning, exactly in the middle, at the end, or randomly placed with padding before, between, and after them. This lets the authors isolate length effects from fact placement.

• The dataset is used as a controlled evaluation set rather than a training set. No training split or mixture ratio is described. The authors first verify that base instances are answerable by testing models on the shortest prompt containing only the question and key paragraphs. Four of five models reach above 0.89 accuracy without chain-of-thought prompting, and the lowest model, GPT-3.5, still reaches 0.77.

• The authors release the dataset and generation code so future evaluations can regenerate fresh instances, supporting studies of long-input reasoning and reducing data contamination risk.

Method

The method is a behavioral evaluation pipeline for measuring how input length affects large language model reasoning over text. The core idea is to control the length of the prompt while keeping the reasoning-relevant evidence unchanged. The evaluation is built from question answering instances that require model to combine information from multiple text spans rather than relying on parametric knowledge or a single isolated sentence.

Each base instance contains three structural components: an optional prefix, two key paragraphs, and an optional suffix. The key paragraphs are the only spans that contain the information needed to solve the task. Each key paragraph is derived from a simple factual sentence and then expanded into a thematically coherent paragraph. The authors use GPT-4 to generate the paragraph expansion under a constraint that no new reasoning-relevant information is introduced, followed by manual verification to preserve the original factual content. The prefix and suffix are also generated to frame the task and pose the question, while the final prompt is assembled by joining these parts with newlines.

The instance design enforces three important properties. First, each task depends on multiple relevant spans that are jointly necessary and sufficient for the answer. This prevents models from solving the task by extracting a single fact. Second, the supporting facts are intended to be novel, reducing reliance on pretraining knowledge. Third, the input is constructed to appear natural, so the background material is organized into coherent paragraphs rather than arbitrary sentences or disjoint fragments.

To isolate the effect of input length, the authors keep the two key paragraphs fixed across length variants and modify only the background text. Three padding strategies are used. In the duplicate condition, the two key paragraphs are repeated in alternating order until the target length is reached. This creates a controlled case where the relevant information is present multiple times, making retrieval easier and isolating whether the model can still use the information when the input becomes long. In the similar condition, padding is sampled from other base instances of the same task. Paragraphs containing entities that also appear in the key paragraphs are excluded to avoid contradictions or ambiguity. This padding resembles a retrieval augmented setting, where the prompt contains several independent but related paragraphs. In the different condition, continuous background text from a book corpus is used, and the key paragraphs are inserted into that text while respecting sentence boundaries. This produces a more varied and natural looking long prompt while keeping the reasoning-relevant spans unchanged.

Overall, the pipeline constructs length-controlled question answering prompts from minimal reasoning atoms. The key paragraphs provide the fixed evidence needed for the task, while the background text controls length under varying degrees of similarity and naturalness. The final prompt is then passed to the model as a standard input, allowing open and closed models to be compared through their observable behavior rather than through access to internal states.

Experiment

The experiments use FLenQA to isolate input length while testing five LLMs on true/false reasoning tasks, starting with baseline settings to confirm that the questions are answerable. They show that accuracy declines as context grows, especially when relevant facts are separated by irrelevant text, and that more realistic irrelevant content can make performance worse rather than easier. Additional analyses find that chain-of-thought only partially helps, next-word prediction does not align with reasoning quality, and longer inputs trigger failures such as refusals, label bias, instruction violations, and missed relevant passages.

At the shortest evaluated input, the models achieve high accuracy across the reasoning tasks, indicating that the task itself is solvable under minimal context. Chain-of-thought prompting improves accuracy for most model-task pairs, especially for weaker or less near-ceiling models, while some already strong models show little change or slight variation. These minimal-length results establish a high baseline against which performance drops with longer inputs can be compared. Minimal 250-token inputs allow the models to perform well on the reasoning tasks. Chain-of-thought prompting generally increases accuracy across models and tasks. GPT-4 is already near ceiling in the Direct setting, leaving little room for improvement. The table supports the broader finding that short-input accuracy does not fully predict robustness to longer context.

The experiments evaluate model performance on reasoning tasks using minimal 250-token inputs to establish a solvability and accuracy baseline. Chain-of-thought prompting generally improves accuracy, particularly for weaker models, while already strong models such as GPT-4 show little gain. These results indicate that strong performance on short inputs does not fully predict robustness as context length increases.