One-sentence Summary

Technical University of Darmstadt, Hessian Center for AI, and MBZUAI propose PeerQA, a real-world scientific document-level QA dataset of 579 pairs sourced from peer reviews and annotated by original authors, which supports evidence retrieval, unanswerable question classification, and answer generation while revealing that decontextualization consistently improves retrieval and establishing a challenging benchmark for long-context modeling on papers averaging 12k tokens.

Key Contributions

• The paper introduces PeerQA, a real-world scientific document-level QA dataset containing 579 QA pairs from 208 academic articles, with questions from peer reviews and answers annotated by original authors, supporting evidence retrieval, unanswerable-question classification, and answer generation.
• Baseline systems and task analyses are established for all three PeerQA tasks, showing that decontextualization consistently improves document retrieval and that answerability models tend to over-answer or under-answer.
• Long-context answer generation is evaluated on papers averaging 12k tokens, where top retrieved passages often outperform full-paper context and better retrieval leads to better generated answers.

Introduction

The exponential growth of scientific publications has increased the burden of literature review, making tools that extract targeted information from papers valuable for researchers and peer reviewers. Question answering systems integrated into reading workflows could help readers locate relevant evidence efficiently and reduce overlooked points in reviews. Prior scientific QA datasets have faced major constraints: expert annotation is hard to recruit, naturally occurring questions are scarce, and many existing datasets rely on templates, search engine suggestions, non expert annotators, or questions that are generic or not grounded in a specific paper. The authors leverage peer reviews to create PeerQA, a real world, document-level QA dataset with questions raised by reviewers and answers annotated by paper authors. It supports evidence retrieval, answerability prediction, and free-form answer generation, and releases 579 annotated samples from 208 papers plus 12k unlabeled questions, while establishing task baselines and analyzing key performance factors.

Dataset

• The authors build PeerQA from papers and peer reviews collected from three source groups: NLPeer, public peer-review venues, and OpenReview.

• NLPeer contributions: ARR 2022, COLING 2020, ACL 2017, CoNLL 2016, and F1000, originally curated in NLPeer and released under CC-BY-NC-SA 4.0.

• Geoscience contributions: papers and reviews from Earth System Dynamics (ESD) and Earth Surface Dynamics (ESurf), released under CC-BY 4.0.

• OpenReview contributions: papers and reviews from ICLR 2022/2023 and NeurIPS Datasets and Benchmarks Track 2022. These are not published in the release because they lack a clear license, so the authors provide download and processing scripts instead.

• The released questions and answers are published under CC-BY-NC-SA 4.0.

• The authors process camera-ready paper PDFs with GROBID 0.8 to extract full text, equations, captions, and paragraph-level sentence groups. These extracted structures are later used to align annotated evidence with the paper text.

• Initial questions are automatically extracted from peer reviews by selecting sentences ending in a question mark, producing 17,910 candidate questions.

• The authors clean and decontextualize these questions with InstructGPT. They use the preceding three review sentences plus the question to rewrite it into a single standalone question. This also reduces noise from review spelling and grammar errors.

• To handle compound or follow-up questions, the authors use a constituency parser to flag questions with root-level conjunctions and then decompose them with InstructGPT.

• The authors manually filter the processed questions to keep only information-seeking questions relevant to the paper. They remove rhetorical questions, typo/layout questions, ambiguous questions, hallucinated content, and questions referring to line numbers or other references missing from the camera-ready version.

• This manual filtering removes 30% of the initial questions and yields a final pool of 12,546 questions.

• Answer annotation is performed by paper authors on the final published version of each paper, not the submitted version. Authors are contacted by email and guided with guidelines, task explanations, and an annotation interface.

• The answer annotation schema has four main parts:

  • Question feedback, where authors can flag or revise a question.
  • Answer evidence, where authors highlight relevant text in the final paper PDF.
  • A free-form answer to the question.
  • An unanswerable flag, used when a question was addressed in rebuttals but not incorporated into the final paper.

• Some questions have only answer evidence or only a free-form answer, but not both.

• The authors map author-highlighted evidence back to the GROBID-extracted paper text. Because GROBID sometimes misses paragraphs needed for alignment, they publish both the raw annotation data and the mapped data.

• As a quality-control step, the authors manually process author feedback on questions and update or remove problematic questions.

• To reduce variance in answer quality, the authors augment collected free-form answers with GPT-4 rephrases while keeping the original annotation spirit.

• The answered subset contains 579 answers from 208 papers. Per-venue annotation counts are reported in Table 2 of the paper.

• The unanswered subset contains 11,967 questions from 2,623 papers, released to support broader question understanding and future annotation work.

• Key dataset statistics reported by the authors:

  • Average paper length is 11,723 tokens.
  • Average question length is 20.2 tokens, longer than average questions in BioASQ, QASPER, and QASA.
  • Processed questions remain highly similar to the original review questions, with 90% above 0.6 similarity and 50% above 0.82 similarity.
  • Each question has 3.8 annotated evidence sentences on average.
  • 30% of questions have non-consecutive answer evidence, meaning evidence is spread across the paper.
  • A sample of 100 questions shows 44% target methods or data clarification, and 12% request justification for a decision.

• The authors use the data as an expert-annotated scientific QA benchmark. The processed full text and paragraphs support evidence-grounded question answering, and the long papers make the dataset useful for long-context models.

• The provided sections describe dataset construction, annotation, quality control, and release details, but they do not specify a training split or mixture ratios.

Method

The authors leverage an end-to-end data curation pipeline that transforms raw peer review text into expert-annotated document-level QA instances. The workflow connects three inputs, the submitted article, the peer review, and the camera-ready publication. Refer to the framework diagram.

In the first module, papers and reviews are collected from multiple scientific venues. For each publication, the full text of the camera-ready version is extracted, including equations and captions, while sentence grouping into paragraphs preserves document structure for downstream evidence modeling. This provides a stable text base for annotations.

The second module performs question extraction and normalization. Review sentences that end with a question mark are collected as candidate questions. Since raw review questions may be noisy, context dependent, or compound, the authors apply a correction procedure. An instruction-following language model receives the three preceding review sentences and the extracted question, then rewrites it into a clean, context-independent query. A constituency parser is then used to detect root-level conjunctions that indicate multiple or follow-up questions, and flagged compounds are decomposed into separate queries. Finally, manual filtering removes rhetorical, layout-related, ambiguous, erroneous, or irrelevant questions, leaving only information-seeking queries that can be matched to the final paper.

The answer annotation module engages the original paper authors as expert annotators. Because questions originate from submitted drafts, answerability depends on whether the corresponding information appears in the camera-ready article. The authors are asked to first validate or revise a question, then judge whether it is answerable from the final publication. If answerable, they highlight answer evidence in the PDF and provide a free-form answer. Unanswerable cases are also permitted, for example when a reviewer issue was addressed in a rebuttal but not incorporated into the final paper.

A quality control layer operates both before and after annotation. Authors can submit feedback through the interface, prompting manual review, correction, or removal of problematic questions. The authors also augment free-form answers by generating rephrased versions with a stronger language model, reducing variance in answer detail and clarity. To support future extraction improvements, both raw annotation mappings and PDF-mapped evidence are released.

Experiment

The experiments evaluate evidence retrieval, answerability, and answer generation for single-paper question answering using several retriever architectures and instruction-tuned LLMs. Retrieval results show that paragraph level retrieval with title augmentation is more effective than sentence level retrieval and provides useful evidence for generation, while answerability results reveal model dependent biases toward predicting answerable or unanswerable questions. Answer generation results indicate that RAG with fewer relevant passages generally works better than full text context, gold evidence serves as an upper bound, and stronger long context models can maintain or improve performance as context length grows.

PeerQA differs from prior scientific QA datasets by sourcing questions from expert peer reviews and answers from paper authors, yielding a small expert-annotated multi-domain dataset with extractive, free-form, and unanswerable answers. Compared with larger or semi-automated resources, it has fewer papers and annotations but stronger annotator expertise and full-paper question context. Experiments show that retrieval-augmented generation with fewer relevant passages often improves LLM answer correctness over full-text settings, though strong long-context models such as GPT-4o remain more stable as context increases. PeerQA uses review-written questions and author-provided answers, grounding supervision in expert sources. It is smaller than several broader scientific QA datasets while retaining free-form and unanswerable answer types. Most evaluated LLMs perform better with retrieved relevant contexts than with full paper text.

PeerQA collects question-answer pairs from multiple machine learning and NLP venues, including recent ML conferences and older NLP events. The venues differ in how many questions receive evidence annotations and free-form answer annotations, with evidence coverage generally present across venues and free-form coverage varying more. Some questions in every venue remain unanswerable because the paper does not provide enough information. Machine learning venues contribute more collected questions than the smaller NLP venues shown Evidence annotations are common across venues, while free-form answer coverage varies by venue Every listed venue includes a subset of questions that cannot be answered from the available paper information

Paragraph-level retrieval generally outperforms sentence-level retrieval for answer evidence, and adding the paper title to paragraphs further improves performance for most models. Sentence-level scores are less affected by title prepending, likely because the title can distort short sentence representations. The strongest models are MiniLM-L12-v2, Dragon+, and SPLADEv3, with SPLADEv3 selected for downstream retrieval-augmented generation due to its high recall. Retrieving paragraphs yields higher answer evidence retrieval quality than retrieving sentences. Prepending the paper title to paragraphs improves results for most models, supporting decontextualization of passages. Title prepending has little effect on sentence-level retrieval because short texts are more sensitive to added context. MiniLM-L12-v2, Dragon+, and SPLADEv3 are the top performers, and SPLADEv3 is chosen for RAG because it achieves the highest recall.

The error analysis of the lowest-scoring generations shows that many cases are not true model failures. A large share of these generations are effectively correct or are penalized by evaluation limitations, such as verbosity or alternative phrasing. The remaining errors involve partial correctness, incorrect reasoning over the evidence, insufficient context or evidence, or problems in the free-form reference answer. Correct answers or evaluation-related issues account for the largest share of low-scoring generations. Partial correctness and reasoning errors occur when the model misses important details or reaches an opposite conclusion. A smaller set of errors stems from insufficient context, insufficient evidence, or inadequate free-form answers.

The experiments evaluate PeerQA, an expert-annotated scientific QA dataset built from peer-review questions and author answers, by comparing retrieval granularities, retrieval models, and retrieval-augmented generation across different paper contexts. Results show that paragraph-level retrieval with paper titles generally outperforms sentence-level retrieval, with SPLADEv3 selected for its strong recall, and that providing fewer relevant passages to LLMs often yields more correct answers than supplying full paper text, although long-context models remain relatively stable. Error analysis indicates that many low-scoring outputs are artifacts of evaluation rather than true failures, with genuine mistakes mainly involving incomplete reasoning or insufficient evidence.