One-sentence Summary

Stanford University researchers introduce SQuAD, a reading comprehension dataset of over 100,000 crowdsourced questions on Wikipedia articles with answers as text segments, and demonstrate a logistic regression model achieving an $F_1$F1​ score of $51.0\%$51.0% (a significant improvement over a $20\%$20% baseline) versus $86.8\%$86.8% human performance, establishing a challenging benchmark.

Key Contributions

• The Stanford Question Answering Dataset (SQuAD) is introduced, consisting of 107,785 crowdsourced question-answer pairs on 536 Wikipedia articles, where each answer is a span of text from the passage.
• Automatic techniques based on distances in dependency trees categorize answer types and stratify questions by difficulty, revealing diverse reasoning patterns.
• A logistic regression model with lexicalized and dependency path features achieves an F1 score of 51.0%, compared to human performance at 86.8% F1, highlighting a challenging benchmark for reading comprehension.

Introduction

Reading comprehension demands that machines understand natural language and world knowledge, but progress has been limited by the available datasets: small, high-quality sets lack the scale needed for modern data-hungry models, while large sets are semi-synthetic and fail to reflect real question-answering patterns. The authors address this gap by introducing the Stanford Question Answering Dataset (SQuAD), a large-scale, human-curated collection of over 100,000 question-answer pairs on Wikipedia articles, where each answer is a span directly extracted from the passage. This design forces systems to select among many candidates without relying on multiple-choice options, and the dataset reveals diverse, stratified question types that remain challenging for both traditional feature-based and neural models, with human performance still well above the best system.

Dataset

The SQuAD dataset is built from English Wikipedia articles and designed for span-based reading comprehension. The authors collect passages, crowd-source question‑answer pairs, and then gather additional answers for evaluation. Here is a quick overview of the dataset composition, processing, and usage.

• Source and passage curation

• Start with the top 10,000 English Wikipedia articles by internal PageRank. Randomly sample 536 articles.

• From each article, extract individual paragraphs, remove images, figures, and tables, and discard any paragraph shorter than 500 characters. This yields 23,215 paragraphs spanning diverse topics.

• Randomly split the paragraphs into training (80%), development (10%), and test (10%).

• Primary question‑answer collection

• Crowdworkers on Amazon Mechanical Turk (via the Daemo platform) are shown one paragraph at a time and asked to write up to 5 questions, then highlight the exact answer span in the paragraph.

• To encourage lexical variety, workers are prompted to use their own words and copy‑paste from the paragraph is disabled.

• Additional answers for robust evaluation

• For each question in the development and test sets, at least two extra crowdworkers provide additional answer spans. They select the shortest span that answers the question, or mark the question as unanswerable (2.6% of questions were marked unanswerable by at least one additional worker).

• This yields multiple ground‑truth answers per question, allowing evaluation with exact‑match and macro‑averaged F1.

• Processing and metadata

• Passages are plain text: all non‑text elements (images, tables, figures) are stripped.

• A syntactic divergence metric (edit distance between unlexicalized dependency paths from a shared anchor to the wh‑word and answer span) is computed to stratify the test set by difficulty in later analysis (Section 6.3).

• Answer types are automatically labeled (date, number, person, location, other proper noun, common noun, adjective phrase, etc.) to provide diversity statistics.

• How the data is used

• The training split is used to train models that predict an answer span given a paragraph and a question.

• The development set is used for model selection, and the test set is used for final evaluation, exploiting the multiple reference answers to compute robust scores.

• The authors also manually label a sample of dev questions with reasoning categories (syntactic divergence, lexical variation, etc.) to analyze model behavior, but those labels are not part of the standard training corpus.

Method

The authors frame answer extraction as a span selection task over individual sentences, leveraging constituency parses to generate candidate answer spans. Rather than considering all $O(L^2)$O(L2) possible spans in a sentence of length $L$L, they restrict candidates to constituents produced by the Stanford CoreNLP constituency parser. In the development set, 77.3% of ground-truth answers align exactly with a constituent; when an answer does not match any constituent, the shortest containing constituent is used as the target during training. This pruning step significantly reduces the candidate space and guides both baselines and the logistic regression model.

As a baseline, a sliding-window approach is employed. For each candidate answer, unigram and bigram overlap between the question and the sentence containing the candidate (excluding the candidate itself) is computed. Candidates with maximal overlap are retained, and the best is selected via the sliding-window method of Richardson et al. (2013), adapted to operate on the sentence level rather than the full passage for efficiency. A distance‑based extension is also implemented.

The core method is a logistic regression model that scores each candidate answer using a rich set of discrete features. Every continuous feature is bucketed into 10 equally sized bins, yielding roughly 180 million features, many of which are lexicalized or derived from dependency tree paths. The feature set includes matching word and bigram frequencies, a root match indicator, length features, span word frequencies, constituent label, and span POS tags. Lexicalized features address lexical variation, while dependency path features handle syntactic variation. The model is trained with a multiclass log‑likelihood loss optimized via AdaGrad (initial learning rate 0.1). For efficiency, updates are performed on batches that group all questions belonging to the same paragraph, since they share candidate spans. An $L_2$L2​ regularization coefficient of $0.1 / \text{number of batches}$0.1/number of batches is used, and the model is trained for three passes over the training data. After training, the candidate with the highest score is selected as the predicted answer span.

Experiment

Evaluation uses exact match and macro-averaged F1, ignoring punctuation and articles; human test performance reaches 77.0% exact match and 86.8% F1, with mismatches mainly arising from non-essential phrase boundary choices. The logistic regression model outperforms baselines but still falls short of human scores, with feature ablation showing that lexicalized and dependency path features are most critical. Model accuracy is highest on dates and numbers and degrades for other answer types and with syntactic divergence, while human performance is unaffected by syntactic divergence, suggesting more robust comprehension.

SQuAD stands out as the largest crowdsourced reading comprehension dataset with open-ended span answers, exceeded in size only by the semi-synthetic cloze-style CNN/Daily Mail corpus. Its free-form answering style is similar to TREC-QA but is implemented as a span prediction task, contrasting with the multiple-choice and sentence-selection formats of other datasets. SQuAD (100K questions) is orders of magnitude larger than all manually curated datasets; only the semi-synthetic CNN/Daily Mail (1.4M) is bigger. While MCTest and Science use multiple choice, SQuAD asks models to extract a specific span, making it more open-ended like TREC-QA's free-form responses.

Answers in the dataset span a wide range of types beyond named entities. Numerical and proper noun answers together account for roughly half the data, while common noun phrases make up the largest single category. The model performs best on dates and numbers, struggles more with named entities, and is weakest on the remaining answer types that form nearly half the dataset, whereas humans maintain high accuracy across all types with only a slight decline on non-entity answers. Common noun phrases and proper noun categories each represent about one-third of answers, while dates and other numeric answers make up roughly one-fifth. The model achieves its highest performance on dates and numbers, where answer candidates are few and often single tokens. Named entities such as person, location, and other entities are moderately challenging for the model but still benefit from clear POS tag features. The remaining answer types, including common noun phrases, adjective phrases, and clauses, together constitute 47.6% of the data and are where the model struggles most. Human performance degrades only slightly on non-entity answers, in contrast to the model's substantial drop, showing that humans are robust to answer type diversity.

Manual labeling of 192 SQuAD questions reveals that every example involves lexical or syntactic divergence between question and answer sentence. Syntactic variation is the most common reasoning challenge, affecting 64.1% of questions, while lexical synonymy appears in 33.3%. This underscores the dataset's emphasis on surface-form mismatches rather than deeper reasoning. Every labeled question exhibits some form of lexical or syntactic divergence between the question and the answer sentence in the passage. Syntactic variation is the most frequent reasoning type, present in 64.1% of the sampled questions.

The logistic regression model combines surface matching features (word/bigram overlap and root match) to identify supporting sentences, length and span frequency features to bias answer span selection, and constituent and POS tag features to steer toward appropriate answer types. Ablation results indicate that lexicalized and dependency tree path features are the most critical to performance, with dependency paths playing an especially large role in this dataset. Matching word frequency, bigram frequency, and root match features help the model select the correct sentence. Length features bias the model toward common answer span lengths and positions, while span word frequencies penalize uninformative words. Constituent label and span POS tag features guide the model toward the correct answer types. Lexicalized features and dependency tree path features are the most important for overall performance, with dependency paths contributing more here than in prior work.

Logistic regression substantially outperforms the baselines, achieving 40.4% exact match on the test set compared to 12.5% for the best baseline, but remains far below human performance of 77.0%. The model identifies the correct sentence 79.3% of the time, indicating that the main challenge lies in extracting the exact answer span. Humans are robust to syntactic divergence, whereas model accuracy degrades as divergence increases. Logistic regression reaches 40.4% exact match and 51.0% F1 on the test set, compared to 12.5% and 19.7% for the strongest baseline. Human test performance (77.0% exact match, 86.8% F1) is markedly higher, with the model's primary difficulty being precise span selection after the correct sentence is found.

The evaluation applies a logistic regression model to SQuAD, a large span-extraction reading comprehension dataset characterized by pervasive lexical and syntactic divergence between questions and passages, with syntactic variation being the dominant reasoning challenge. Ablation studies show that dependency path and lexicalized features are most critical for performance, and while the model substantially outperforms a sliding-window baseline, its main shortfall is extracting the precise answer span after correctly identifying the supporting sentence. Model accuracy varies sharply by answer type, performing best on dates and numbers but struggling with common noun phrases and other non-entity answers, whereas human performance remains robust across all categories.