One-sentence Summary

Researchers at the University of Texas at Austin and BespokeLabs.ai introduce QSALIENCE, an instruction-tuned salience predictor for inquisitive questions trained on a linguist-annotated dataset of 1,766 $(context, question)$(context,question) pairs, showing that highly salient questions are more likely to be answered in the same article and serve as an indicator of news summarization quality.

Key Contributions

• QSALIENCE is introduced as an instruction-tuned salience predictor for inquisitive questions, trained on 1,766 linguist-annotated context-question pairs scored on a 1-5 Likert scale.
• Higher salience scores are shown to predict whether inquisitive questions are answered later in the same article, providing an empirical link between potential questions and Questions Under Discussion.
• Downstream experiments connect salience to summary expansion and news summarization: summaries answering more salient questions are judged higher quality, and fine-tuned salience prediction models outperform GPT-4 on salience prediction.

Introduction

Pre-trained language models can now generate open-ended, curiosity-driven questions from text, supporting tasks such as reading comprehension, elaboration, news perspective analysis, follow-up questions, and summarization. The challenge is that many possible questions can be generated from the same context, yet only some are useful for understanding the discourse; prior linguistic and computational work has not sufficiently modeled which questions readers expect to be answered or how salient they are. The authors introduce a linguist-annotated corpus of inquisitive questions with salience and answerability ratings, showing that salience aligns with answer utility and known Questions Under Discussion. They then train QSALIENCE, an instruction-tuned model for predicting question salience from article context, which outperforms GPT-4 prompting and supports smaller models. They also demonstrate that expanding summaries to answer more salient questions improves reader rankings.

Dataset

• The authors introduce QSALIENCE-data, a corpus of 1,766 inquisitive questions with salience annotations and natural language rationales for each judgment.

• The corpus is assembled from multiple existing text sources to support different analyses:

  • DCQA: news articles with LLM-generated questions and existing annotated QUDs, used to compare general inquisitive questions with QUDs.
  • TED-Q: TED talk excerpts with human-generated questions, used for salience and answerability analysis.
  • DiverseSumm: newer news articles used for downstream summarization. The authors denote the full article subset as Div.Article and the generated short summary subset as Div.TL;DR.

• The reported source subset details are:

  • 1,705 machine-generated questions, created using four models: 250 from Llama-2-7B-chat, 249 from Mistral-7B-instruct, 100 from GPT-3.5-turbo, and 1,106 from GPT-4-turbo.
  • 61 human-generated questions from prior corpora: 36 from two DCQA articles and 25 from one TED-Q article.
  • DiverseSumm articles are selected to be roughly 1,500 words long for fair downstream evaluation.
  • Div.TL;DR contains GPT-4-turbo generated summaries of about 50 words, typically three sentences, for which question salience is also annotated.

• Question construction uses an anchor-sentence and context window approach:

  • The prompt gives the LLM a preceding context, denoted as Cp, and an anchor sentence, Sk.
  • The model is asked to generate five questions about a part of the anchor sentence that a reader may be curious about.
  • For full articles, question generation probes begin at the 4th sentence and continue through the 16th sentence, with two-sentence gaps between probes.
  • For short TL;DRs, questions are generated per sentence.

• Salience annotation and filtering:

  • Three trained undergraduate annotators provide salience labels and natural language rationales for each question.
  • More than 90.8% of LLM-generated questions are marked valid.
  • Invalid questions commonly lack the correct anchor sentence or contain nonfactual presuppositions.
  • Salience labels are aggregated by averaging the annotator scores and rounding to the nearest integer.
  • Inter-annotator agreement is reported as substantial, with Krippendorff’s alpha values of 0.719 for DCQA, 0.632 for TED-Q, 0.751 for Div.Article, and 0.649 for Div.TL;DR.

• The dataset schema includes question text, source article or summary, preceding context, anchor sentence, salience label, rationale, question validity, and the LLM that generated the question when applicable.

• A separate answerability annotation subset is constructed for analysis:

  • 225 valid DCQA questions and 86 valid TED-Q questions are annotated for whether the question is answered in the following discourse.
  • The answerability scale is: 0 already answered in the preceding context, 1 unanswered, 2 partially answered, and 3 fully answered.
  • Inter-annotator agreement for answerability is 0.799.

• The paper uses the dataset to analyze whether salient potential questions are also answered later in discourse, to compare QUDs with random potential questions, and to build and evaluate salience prediction settings.

• Annotator rationales are used for few-shot Chain-of-Thought prompting, and valid annotated questions provide the supervised signal for salience prediction.

• No explicit training split or mixture ratio is specified in the provided text; the descriptions focus on annotation pools, source subsets, and prediction use cases.

• In the downstream summarization use case, the authors sample 34 DiverseSumm articles of about 1,500 words and generate GPT-4-turbo TL;DRs of about 50 words.

• They also create three expanded summary conditions with lengths controlled to 230 to 250 words:

  • GPT-4 expanded summaries.
  • Flan-T5-large expanded summaries, refined with GPT-4-turbo to correct obvious errors while preserving the original summary.
  • GPT-4-corrupted summaries, synthesized to omit prevalent important topics and then refined for coherence and length.

• Each sentence in every TL;DR is associated with five LLM-generated inquisitive questions, annotated with salience.

• The fine-tuned salience models are then applied to all valid questions in the downstream set to produce predicted salience values.

Method

The core method focuses on predicting the salience of valid questions by treating the task as a discourse-level scoring problem. For each valid question, the model receives a structured input consisting of the parent context $C_p$Cp​, the target sentence span $S_k$Sk​, and the question $Q_*$Q∗​. The prediction target is the salience score associated with the question given that context. This formulation allows the model to recover implicit discourse information, such as whether a question targets a key, informative, or prominent part of the surrounding text rather than a peripheral or underdeveloped detail.

The authors build an instruction-tuned salience predictor, referred to as QSALIENCE, by applying QLoRA to several open-source large language models, including Mistral-7B-Instruct, Llama-2-7b-chat, TinyLlama-1.1B-chat, and Flan-T5-base. QLoRA enables efficient low-rank adaptation of large language models, making it possible to fine-tune them on the salience prediction task without full-parameter training. Optimization is performed with AdamW, and the task is cast as supervised sequence-to-sequence or instruction-following learning, where each training instance is represented as an input-output pair:

For most models, the instruction prompt frames the task similarly to the human annotation guidance. For Flan-T5-base, the available context window is only 512 tokens, so the model is additionally tested without explicit instructions. In that setting, the context is truncated in reverse sentence order from $C_p$Cp​ and $S_k$Sk​ until the maximum length is filled, preserving the most locally relevant content while fitting the sequence budget.

To establish strong comparison points, the method also includes zero-shot and few-shot prompting pipelines using GPT-4-turbo. In the zero-shot vanilla setting, the model receives an instruction that asks it to assign a salience score, closely mirroring the criteria given to human annotators. In the few-shot vanilla setting, the prompt contains 15 in-context demonstrations, three for each salience label. Each demonstration maps a contextual input to a salience score. The ordering of these demonstrations is deliberately arranged so that examples with more frequent training labels appear near the end of the prompt. This design exploits the recency bias of large language models to encourage predictions that better match the label distribution of the task.

A k-nearest-neighbor few-shot variant is also introduced to make demonstration selection more instance-dependent. For each test instance, the parent context $C_p$Cp​ and sentence span $S_k$Sk​ are encoded separately with RoBERTa-large. The CLS embeddings for the two components are averaged to form a single representation. Nearest-neighbor examples are then retrieved using Euclidean distance, with one closest example selected for each salience label. These retrieved demonstrations are ordered in the prompt using a strategy similar to the vanilla few-shot setting, preserving both task distribution and instance-specific relevance.

The prompting strategy is further extended with chain-of-thought prompting. In the few-shot chain-of-thought configuration, the prompt includes 5 examples that contain natural language rationales drawn from human salience annotations. This encourages the model to generate intermediate reasoning before producing the final salience judgment, testing whether explicit justification improves agreement with human annotations.

For the downstream summary quality analysis, the salience predictor is integrated into a generated summary pipeline. Source articles are first compressed into short TL;DR summaries, for example 50-word summaries produced with GPT-4-turbo. These summaries are then used as the basis for generating longer expanded summaries, typically between 230 and 250 words. Three expanded summary variants are constructed. The first is generated directly by GPT-4-turbo given the source document and the TL;DR. The second is generated by Flan-T5-large and then refined by GPT-4-turbo to correct obvious problems such as repeated sentences or failure to follow length constraints. The third is a synthetically corrupted summary generated by asking GPT-4-turbo to produce a longer summary that omits important topics identified from the TL;DR; the corrupted output is also refined to improve coherence and length compliance while preserving the intentionally reduced topical coverage.

For each TL;DR, every sentence is associated with generated inquisitive questions, and each valid question is scored for salience. Human annotation and model prediction are both used: the questions receive salience labels, and the fine-tuned salience models are run over the valid questions to obtain predicted salience values. This design allows the system to compare whether stronger expanded summaries tend to answer questions with higher predicted salience, while the corrupted summary baseline provides a controlled negative signal for summaries that omit central topics.

Experiment

The experiments evaluate whether annotated question salience tracks answerability, whether models can predict salience, and whether salience-based scores reflect summary quality. Annotated potential questions that are more salient are also more likely to be answered later, indicating alignment between reader curiosity and discourse development. For salience prediction, fine-tuned models learn the notion better than prompted large language models and achieve moderate agreement with human judgments, though the task remains challenging. In the summary use case, aggregating the salience of questions answered in expanded summaries reproduces human quality rankings, supporting the idea that better summaries address more salient questions.

The table summarizes a multi-source question collection drawn from news articles, TED talks, and DiverseSumm material. The DiverseSumm article and TL;DR subsets provide the largest share of articles and questions, while the TED-Q subset is much smaller. Average question lengths are modest and broadly comparable, with the TL;DR subset having the longest typical questions and moderate variation across datasets. The dataset mixes human annotated and machine generated questions from multiple corpora. DiverseSumm contributes the largest amount of data for downstream salience analysis. Question length trends are consistent across subsets, with only the TL;DR subset standing out as slightly longer.

Human annotation shows that most generated questions are judged valid, indicating that language models can produce promising inquisitive questions. The annotated salience labels are spread across the scale rather than concentrated at low levels, and the distribution varies by source dataset. This variation supports using salience as a useful signal for downstream question evaluation and expansion tasks. Most questions are labeled valid, with invalid questions being the exception across datasets. Valid questions with the lowest salience are rare, suggesting that valid questions are generally not irrelevant to the surrounding context. Salience distribution differs by dataset, with diverse TLDR questions showing a stronger concentration toward higher salience scores. The spread of salience labels indicates that not all valid questions are equally useful, making a salience predictor potentially valuable.

The annotated answerability distribution differs by data source, with TED-Q questions showing the largest share in the fully answered category. DCQA questions are more concentrated in lower answerability categories, especially already answered and unanswered by the later context. TED-Q questions are more often judged fully answered in the following context than DCQA questions. DCQA questions have a larger share in lower answerability categories, including already answered and unanswered by the later context.

Human ratings indicate that more salient potential questions tend to be more answerable in later discourse, substantially exceeding a random baseline. This positive association appears in both annotated subsets, with TED-Q showing a stronger relationship than DCQA. The pattern suggests that reader salience and writer answerability are related. Salience and answerability are positively related in the annotated human judgments. The relationship is stronger for the TED-Q subset than for the DCQA subset. A random baseline shows no comparable association.

Compared with randomly sampled potential questions, human annotated QUDs are distributed toward higher salience scores, with fewer low salience cases and more high salience cases. This pattern suggests that questions likely to be answered later in the discourse are also more likely to be salient potential questions. QUDs show a stronger concentration in the higher end of the salience distribution than random potential questions. The low end of the salience distribution is much less common for QUDs than for the random comparison set. The observed shift supports the idea that salience is associated with later answerability.

The experiments build a multi-source question collection and use human annotations to examine question validity, salience, and answerability. The annotations show that most machine-generated questions are valid, while salience and answerability distributions differ across corpora and provide useful signals rather than uniform low-score patterns. Human ratings further support the conclusion that salience is positively associated with later answerability, with human QUDs tending to occupy the higher end of the salience distribution compared with random potential questions.