One-sentence Summary

Stanford University presents SQuAD, a reading comprehension dataset of $100{,}000+$100,000+ crowdworker-posed questions over Wikipedia articles with span-based answers, in which a strong logistic regression model achieves $51.0\%$51.0% F1 against $86.8\%$86.8% human performance, highlighting a substantial challenge for future machine comprehension research.

Key Contributions

• The paper introduces SQuAD, a freely available reading comprehension dataset with 107,785 crowdsourced question-answer pairs over 536 Wikipedia articles, where each answer must be a text span from the corresponding passage rather than a selected answer choice.

• Automatic analysis techniques based on dependency and constituency trees quantify question and answer diversity, characterize the reasoning required by SQuAD, and examine difficulty through answer-type complexity and syntactic divergence.

• A logistic regression baseline achieves 51.0% F1, substantially improving over a 20% sliding-window baseline, while the 86.8% human performance identifies SQuAD as a challenging benchmark for future reading comprehension models.

Introduction

The paper targets machine reading comprehension, where a system reads a passage and answers natural-language questions. This capability matters for applications that require precise understanding of text and real-world knowledge, such as question answering, information extraction, and automated tutoring. Earlier reading comprehension resources faced a tradeoff: high-quality datasets were too small for modern data-intensive models, while larger datasets were often semi-synthetic and unlike realistic explicit reading questions. The authors address this gap by introducing the Stanford Question Answering Dataset v1.0, SQuAD, a large crowd-sourced dataset built from Wikipedia articles with 107,785 question-answer pairs on 536 articles. In SQuAD, systems must answer by selecting a text span from the passage rather than choosing from listed options, making the task both more realistic and easier to evaluate. They also introduce a logistic regression baseline and analyze question difficulty, showing substantial room for improvement over human-level performance.

Dataset

• The authors use SQuAD, a human-authored reading comprehension dataset built from Wikipedia. It combines curated passages, crowdworker-written questions, and answer spans highlighted directly in the passage.

• Source pipeline: they rank English Wikipedia articles with internal PageRank, select the top 10,000, uniformly sample 536 articles, and extract individual paragraphs. They remove images, figures, and tables and discard paragraphs shorter than 500 characters, yielding 23,215 paragraphs across many topics.

• Split design: they partition the 536 articles randomly into 80% training, 10% development, and 10% test. This corresponds to roughly 18.6k training paragraphs, 2.3k development paragraphs, and 2.3k test paragraphs.

• Question creation: qualified crowdworkers with a 97% HIT acceptance rate, at least 1000 completed HITs, and a US or Canada location are paid $9 per hour to ask and answer up to 5 questions per paragraph. Workers are encouraged to ask questions in their own words, copy-paste is disabled on the passage, and interface examples show good and bad question-answer behavior.

• Schema: each example consists of a passage paragraph, a natural language question, and the answer as an exact highlighted span within the paragraph. The answer is not restricted to a single named entity and may be a date, number, proper noun, common noun phrase, adjective phrase, verb phrase, clause, or other text span.

• Answer evaluation augmentation: for development and test questions, the authors collect at least two additional answers. Annotators select the shortest span answering the question or mark the question unanswerable. They report that at least one additional annotator marks 2.6% of questions unanswerable, which helps make evaluation more robust.

• Answer metadata: they classify development-set answers using Stanford CoreNLP constituency parses, POS tags, and NER tags. The answer distribution is 19.8% dates and numbers, 32.6% proper nouns, 31.8% common noun phrases, and 15.8% other types such as adjective phrases, verb phrases, and clauses.

• Difficulty metadata: they compute syntactic divergence between the question and the sentence containing the answer by extracting dependency paths around shared anchors and measuring minimum edit distance. This divergence is used to characterize question difficulty and stratify dataset analysis.

• Model usage: the training split provides paragraph-question-answer-span examples for learning answer extraction. The development and test splits are used for evaluation, and the additional human answers support more reliable baseline comparisons and unanswerable-question checks.

Method

The authors formulate paragraph-based question answering as a candidate generation and ranking problem. Instead of considering every possible text span in a sentence, the pipeline first restricts the search space to linguistically plausible spans. A constituency parser is used to identify constituent spans, and only these constituent spans are treated as candidate answers. This avoids the full quadratic span space of $O(L^2)$O(L2), where $L$L is the number of words in the sentence, and reduces the problem to scoring a smaller set of syntactically grounded answers.

This candidate-generation choice introduces a structural constraint that can also serve as an upper bound on performance. Ignoring punctuation and articles, the authors observe that 77.3% of the correct answers in the development set are constituents. During training, when the correct answer is not itself a constituent, the method uses the shortest constituent that contains the correct answer as the supervision target. This keeps the training signal compatible with the parser-based candidate space.

The main ranking model is a multiclass logistic regression system. For each candidate answer, the authors extract a rich feature representation that combines lexical matching, positional information, syntactic type, word-frequency statistics, and dependency-path patterns. Matching word frequencies, matching bigram frequencies, and root-match features help identify sentence-level evidence relevant to the question. Length-related features bias the model toward answer spans that occur at common lengths and positions, while span word-frequency features reduce the influence of uninformative high-frequency words. Constituent label features and span part-of-speech features guide the model toward answer types that are compatible with the question.

To handle surface variation, the feature set includes lexicalized features that help resolve differences in word choice. Syntactic variation is captured through dependency tree path features, which encode relational patterns between candidate nodes and question-related terms. For example, paths involving dependency relations such as subject or modifier links can help the model recognize whether a candidate participates in the same syntactic relation as the answer in a supporting context. Continuous features are discretized into ten equally sized buckets, resulting in a very large feature space of approximately 180 million features, most of which are lexicalized or dependency-path features.

Given a question $q$q and its candidate set $\mathcal{C}_q$Cq​, the logistic regression model assigns a score to each candidate $c_i$ci​ based on its feature vector $\phi(q, c_i)$ϕ(q,ci​). The score can be written as

and the candidate distribution is obtained through a softmax over the candidate set:

Training maximizes the multiclass log-likelihood of the correct answer under this distribution. The optimization uses AdaGrad with an initial learning rate of $0.1$0.1. For efficiency, each update is performed on the batch of all questions within the same paragraph, since those questions share the same candidate set. This shared-candidate batching allows the model to update all relevant question-candidate interactions in one step.

Regularization is important because the discretized feature space is extremely large. The training objective includes an $L_2$L2​ penalty with coefficient $0.1$0.1 divided by the number of batches, which scales the regularization strength with the size of the training set. The model is trained with three passes over the training data.

The same candidate-answer framework is also used for baseline comparisons. One baseline computes unigram and bigram overlap between the question and the sentence containing a candidate, excluding the candidate itself. Candidates with maximal overlap are retained, and among those, the best one is selected using a sliding-window scoring approach. A distance-based extension is also considered, but it uses only the sentence containing the candidate rather than the full passage, making the baseline more efficient while preserving local context.

Experiment

The experiments evaluate question answering models using exact match and token-level F1, with human performance measured on the development and test sets as a reference for the difficulty of extracting concise answer spans. The main model evaluation shows that the logistic regression system improves over baselines but remains below humans, with ablations indicating that lexicalized and dependency path features are most influential and that the model is strongest when answer candidates are limited and single token. Further analysis by answer type and syntactic divergence reveals that the model struggles more with diverse named entities and superficially different questions, whereas humans are largely insensitive to such surface variation.

The surveyed datasets cover a range of reading comprehension and question answering styles, from multiple choice and sentence selection to open-ended answer spans. The highlighted dataset is notably large for crowdsourced span extraction and is more open-ended than most earlier datasets. Further analysis suggests that identifying the answer sentence is easier than extracting the exact span, especially when questions and answer sentences differ syntactically or when many plausible candidates exist. The dataset is a large crowdsourced reading comprehension resource focused on selecting answer spans rather than choosing fixed answer options. It is more similar to open-ended question answering datasets than to many smaller multiple choice or computation-oriented datasets. Human performance exceeds the logistic model, and most errors involve small wording differences rather than completely wrong answers. The model can often find the correct sentence but struggles to locate the precise answer span inside it. Lexicalized and dependency-tree path features are particularly important for model performance. Model accuracy decreases as syntactic divergence between the question and answer sentence increases, while human performance is less affected. The model works best on answer types with few plausible candidates, such as dates and numbers, and performs worse on more ambiguous categories.

The dataset contains a broad mix of answer types, including numeric expressions, proper noun entities, common noun phrases, and less standard linguistic forms. Performance analysis shows the model does well when candidates are constrained, such as dates and numbers, but struggles more with diverse named entities and especially other answer types that cover nearly half the data. Humans remain strong across numeric and named entity categories, with only slight degradation on other answer types. Numeric and date answers form a substantial share of the dataset. Proper nouns are split into person, location, and other entities rather than treated as one class. The model is strongest on dates and numbers, where plausible answer candidates are limited. Other answer types, including adjective phrases, verb phrases, clauses, and other forms, are hardest for the model.

The manual labeling shows that answering the sampled questions can require different kinds of reasoning, and every example shows some lexical or syntactic difference between the question and the answer sentence. Syntactic variation is the most common category, while lexical variation based on synonymy is also frequent. A smaller number of examples require world knowledge, reasoning across multiple sentences, or are ambiguous, and some examples belong to more than one category. Syntactic variation is the most frequent reasoning category Synonym-based lexical variation is also common, while multiple sentence reasoning and ambiguity are less common

The figure illustrates how syntactic divergence between a question and its answer can be quantified by comparing their dependency-tree paths. The example aligns the question path to the answer path using deletion, substitution, and insertion operations, resulting in a nonzero edit cost. This type of superficial syntactic difference is presented as a source of model difficulty, while human performance appears less affected. Question and answer sentences are compared through dependency-tree paths. Path alignment is measured using edit operations such as delete, substitute, and insert. Higher syntactic divergence corresponds to lower model performance, while humans are not clearly sensitive to it.

The logistic regression model represents candidate answers using discretized lexical, structural, length, frequency, and syntactic features. Matching word and bigram frequencies, root matches, and span properties help distinguish correct answer-bearing sentences and spans. Lexicalized features and dependency path features are especially important for handling lexical and syntactic variation. Matching word frequencies, bigram frequencies, and dependency root matches support selecting sentences that contain the answer. Lexicalized features and dependency tree path features play a major role in the model's performance.

Experiments evaluate a logistic regression model and human performance on a large crowdsourced reading comprehension dataset for answer span selection, analyzing dataset diversity, answer types, reasoning categories, syntactic divergence, and model features. The results show that identifying the answer sentence is easier than extracting the exact span, and the model performs better on constrained answer types such as dates and numbers while struggling with diverse named entities and other answer forms. Humans remain more robust than the model to lexical and syntactic variation, especially as syntactic divergence between the question and answer sentence increases, while lexicalized features and dependency path features are important for model accuracy.