One-sentence Summary

NVIDIA releases HelpSteer2, a permissively licensed ($\text{CC-BY-4.0}$CC-BY-4.0) preference dataset of only ten thousand response pairs, and proposes $\text{SteerLM 2.0}$SteerLM 2.0, a model alignment method that leverages its rich multi-attribute reward scores to achieve a state-of-the-art $92.0\%$92.0% on Reward-Bench's primary dataset, outperforming existing open and proprietary models.

Key Contributions

• The paper releases HelpSteer2, a CC-BY-4.0-licensed preference dataset of only ten thousand response pairs with high annotation quality (Cohen's κ of 0.791), along with detailed documentation of the data collection process to support similar efforts.

• Reward models trained on HelpSteer2 achieve a state-of-the-art 92.0% score on RewardBench's primary dataset as of June 12, 2024, outperforming all open and proprietary models listed there at that time.

• The paper introduces SteerLM 2.0, an alignment approach that effectively exploits the multi-attribute score predictions from the trained reward models, and shows that aligning Llama 3 70B Base with this method matches or exceeds the performance of Llama 3 70B Instruct and GPT-4-0613 on MT Bench, TruthfulQA, AlpacaEval 2.0 LC, and Arena Hard.

Introduction

The authors tackle the growing need for transparent, reusable preference data in large language model alignment. While human preference feedback has proven essential for training capable models, most leading systems only vaguely disclose their data, and existing public datasets either carry restrictive licenses or have become outdated for state of the art alignment. To fill this gap, they introduce HelpSteer2, a permissively licensed CC BY 4.0 dataset with 10,000 high quality preference pairs, along with a detailed collection process to support similar efforts. They also propose SteerLM 2.0, a novel alignment method that leverages multi faceted reward signals to help models follow complex instructions. Their trained reward model on Llama 3 70B demonstrates that this compact dataset can align a base model to match or exceed the performance of Llama 3 70B Instruct and GPT-4 0613 across major evaluation benchmarks.

Dataset

The authors built HelpSteer2, a dataset for reward model training, with a focus on diversity and quality. Here is how the data is composed, processed, and used.

Dataset composition and sources

• Prompt sources: Over 95% of prompts come from ShareGPT, using only user turns (assistant turns are stripped to avoid licensing issues). The remaining prompts are proprietary, targeting enterprise use cases like summarization, closed QA, and extraction.
• Response sources: Each prompt has two responses, always from two different sources. Sources include internal LLMs (Nemotron-2, Nemotron-3, Nemotron-4), Mixtral-8x7B-Instruct-v0.1, and human annotators. Proportions are: 18.9% Nemotron-2 (43B), 40.4% Nemotron-3 (2.2% from 8B, 38.2% from 22B), 26.9% Nemotron-4 (9.5% from 15B, 17.4% from 340B), 7.9% Mixtral, and 5.9% human.

Key details for each subset

• Filtering rules:
  • Removed non-English prompts using FastText (annotators are US-based English speakers).
  • Filtered out prompts with code snippets using simple heuristics (annotators lack coding expertise).
  • Used BERTopic to cluster prompts into ~1000 topics and sampled uniformly across topics.
  • Assessed prompt complexity with Nemotron-2-43B on a Likert-5 scale, then sampled uniformly across levels, except the highest complexity level got double weight.
• Multi-turn prompts: ~29% of samples are multi-turn (mean 2.83 turns). For these, original ShareGPT assistant responses were replaced with responses from an in-house 22B model, fine-tuned on Open Assistant and HH-RLHF.
• Response generation: Two responses per prompt generated by diverse models, intentionally varying in size and training approaches (SFT, SteerLM, RLHF, DPO) to increase diversity. Some responses used random SteerLM labels for style variation.

Annotation and processing

• Attributes: Each response is annotated for helpfulness, correctness, coherence, complexity, and verbosity on a Likert-5 scale.
• Annotators: ~1,000 US-based annotators (vs. 200 in HelpSteer). Each sample gets at least 3 annotators; if disagreement on helpfulness exceeds 2 points, 2 more annotators are added (average 3.41 per sample). Responses are rated sequentially for calibration.
• Quality control: Inter-annotator agreement measured with quadratic weighted Cohen’s κ. Improved from initial κ=0.465 to 0.706 for helpfulness through guideline clarifications and reviewer filtering. Retained only responses where helpfulness differences among annotators are ≤2 points, removing ~10% of samples. Ultimately, ~50% of all annotations were excluded.
• Final dataset: 21,362 high-quality samples, consisting of 10,681 prompts each with two responses. Split into 95% training and 5% validation.

Usage in the model

• The authors use the training subset to train reward models. The dataset is designed to improve reward prediction for multi-turn conversations and to emphasize helpfulness as the primary signal, with correctness as a strong predictor (Pearson’s R=0.9430 vs. 0.8525 in HelpSteer). Coherence is a weaker predictor now (0.4979 vs. 0.6348). Prompt length and turns show slight negative correlation with helpfulness, while response length has a slight positive correlation.

The dataset also shows longer responses (average 1492.6 vs. 497.3 characters) and shorter, more conversational prompts (712 vs. 2491 characters) compared to HelpSteer, reflecting the shift from enterprise-heavy content to ShareGPT-based prompts.

Method

The authors present HelpSteer2, a multi-attribute annotated preference dataset, along with a regression-based reward model and several downstream alignment approaches built on it. The full pipeline spans dataset collection, reward model training, and the alignment of language models through supervised fine-tuning, direct preference optimization, proximal policy optimization, and an improved attribute-conditioned alignment method termed SteerLM 2.0.

Dataset Collection Pipeline

For dataset construction, the authors source prompts primarily from ShareGPT (over 95%), relying exclusively on user inputs to avoid model-specific licensing restrictions, and supplement them with proprietary prompts targeting enterprise use cases such as summarization, closed question answering, and extraction. The filtering stage removes non-English prompts using FastText and filters out coding-related prompts with simple heuristics, since the annotator pool consists of US-based English speakers without coding expertise. To ensure diversity, the authors use BERTopic to cluster prompts into roughly 1000 topics and sample uniformly from each cluster. Prompt complexity is scored on a Likert-5 scale using a Nemotron-2-43B model, and prompts are sampled uniformly across complexity levels, with the highest complexity level weighted twice as heavily. Multi-turn prompts constitute approximately 29% of the samples, with assistant turns generated by a 22B in-house model fine-tuned on Open Assistant and HH-RLHF conversations.

Two responses are generated per prompt, rather than four as in the original HelpSteer, to reduce annotator cognitive load, and the two responses always come from different sources. Response sources span multiple generations of internal LLMs (Nemotron-2 43B, Nemotron-3 8B/22B, Nemotron-4 15B/340B), Mixtral-8x7B-Instruct, and human annotators. All internal models are fine-tuned on permissively licensed data using a variety of techniques, including supervised fine-tuning, SteerLM, reinforcement learning from human feedback, and direct preference optimization, which substantially increases response diversity compared to the original HelpSteer dataset.

For annotation, each response is scored on five Likert-5 attributes: helpfulness, correctness, coherence, complexity, and verbosity. At least three annotators rate each response, with two additional annotators recruited if the disagreement in helpfulness among the initial three exceeds 2 points. Annotators rate the two responses to a common prompt sequentially, which provides more calibrated scores by enabling direct relative comparison. Inter-annotator agreement is measured using quadratic weighted Cohen's κ, chosen because the large annotator pool (approximately 1000 annotators) rarely yields common sets of samples across multiple annotators, making multi-annotator metrics such as Fleiss' κ or Krippendorff's α less applicable. Through iterative clarification of annotation guidelines, agreement for helpfulness improves from an initial 0.465 to 0.706. Only samples where the difference in helpfulness scores among the three retained annotators is 2 points or below are kept, and approximately 50% of all annotations are excluded through multiple filtering stages, yielding a final dataset of 21,362 samples (10,681 prompts each paired with two responses).

Reward Model Training

The reward model consists of a base model augmented with a linear layer that maps the final layer representation at the end-of-response token into five scalar values, one per HelpSteer2 attribute. The authors train this architecture on top of two base models: Llama 3 70B and an in-house Nemotron-4 340B model. Training runs for two epochs with a global batch size of 128, using an MSE loss with a constant learning rate (2e-6 for the 70B model and 7e-7 for the 340B model) and the AdamW optimizer with 10 warmup steps.

The key design choice is the use of regression over fine-grained attributes rather than binary preference classification. A Bradley-Terry-style preference reward model produces rewards that are only comparable within the same prompt, since any prompt-dependent offset can be added without changing the induced preference ordering. This makes cross-prompt comparison ambiguous: a response with reward 4 on one prompt is not necessarily better than a response with reward 2 on another. In contrast, the SteerLM regression reward model predicts absolute scalar ratings for each attribute, enabling consistent comparison across prompts and allowing the model to disambiguate attributes such as verbosity from overall quality. This is particularly important because binary-trained models can spuriously associate longer responses with higher quality, whereas explicit verbosity prediction prevents such confounds.

Supervised Fine-Tuning

The authors first train an SFT model on the Open Assistant dataset (56k conversations) for 2400 steps with a constant learning rate of 2e-6 and batch size 128. This model proves weaker than the Llama 3 70B Instruct baseline, which the authors attribute to inconsistent response quality in Open Assistant. They therefore construct a second SFT model using the "Daring Anteater" dataset, containing 100k conversations averaging 2.88 model turns each. Approximately 93% of this data is synthetically generated by replacing the original generation pipeline's OpenAI models with an earlier aligned version of Nemotron-4 340B and Mixtral-8x7B-Instruct-v0.1, while the remainder is sourced from specialized datasets covering reasoning, mathematics, coding, and table-based QA. This model is trained for 1600 steps with a larger global batch size of 384, and serves as the starting point for all subsequent DPO and PPO experiments.

Direct Preference Optimization

For standard DPO, the HelpSteer2 training set is converted into a preference dataset by selecting the response with the higher helpfulness score as the chosen response and the lower-scoring response as the rejected one, discarding pairs with tied helpfulness scores. This yields 7,221 preference pairs. The SFT model is trained on these pairs for 7 epochs with a constant learning rate of 2e-7, a KL penalty of 1e-3, and weight decay of 0.1.

Iterative DPO then improves the model further by sampling 10 responses per prompt from 20k prompts in the Daring Anteater SFT dataset (temperature 0.7, top-p 0.9), scoring these responses with the Llama 3 70B reward model, and constructing pairwise preference data from the highest and lowest goodness scores. The goodness score is defined as a weighted combination of attributes:

This weighting was found to give the best differentiation between chosen and rejected responses on RewardBench prompts. The iterative DPO training runs for 3 epochs with a KL penalty of 1e-3 and a learning rate of 9e-8.

Proximal Policy Optimization

For PPO, the SFT model serves as the initial policy, and the Llama 3 70B reward model provides the reward signal. The reward is the goodness score normalized by subtracting the mean and dividing by the standard deviation of the HelpSteer2 responses, which stabilizes the reward scale. PPO training uses a global batch size of 128 with a rollout buffer of 128, a constant learning rate of 1e-7, and a KL penalty of 3e-3 over 64 steps. The generation stage is optimized through NeMo-Aligner's integration with TensorRT-LLM for efficient rollout.

SteerLM and SteerLM 2.0

SteerLM aligns language models by conditioning generation on desired attribute values. The original SteerLM approach annotates the Daring Anteater SFT dataset using the reward model and then performs attribute-conditioned supervised fine-tuning, teaching the model to generate responses conditioned on target attribute scores. However, the original formulation does not explicitly enforce that generated responses follow the desired attribute distribution. SteerLM 2.0 addresses this limitation by iteratively training the model to approximate the optimal SteerLM policy derived from the reward model.

The optimal conditional distribution is derived via Bayes' rule. Given a probabilistic reward model $P(a|x,y)$P(a∣x,y) that predicts attribute values $a$a from prompt $x$x and response $y$y, and an unconditional response distribution $P(y|x)$P(y∣x) from a separate language model, the optimal conditional distribution is:

To convert the regression reward model into a probabilistic one, the authors use a Beta distribution to estimate the probability of each reward level. The reward model output $r$r is scaled, and Beta distribution parameters are set as $\alpha = 24r$α=24r and $\beta = 24 - \alpha$β=24−α, where $\alpha + \beta = 24$α+β=24 matches the ground truth distribution of the training data. The probability of a particular attribute level $n$n is computed as the difference between cumulative Beta distribution values at adjacent bin boundaries.

The parametric model $Q_\theta(y|a,x)$Qθ​(y∣a,x) is trained to approximate the optimal $P(y|a,x)$P(y∣a,x) by minimizing the KL divergence:

Expanding this objective reveals that direct optimization requires sampling from the intractable optimal distribution. The authors therefore use importance sampling from the initial SteerLM model $Q'(y|a,x)$Q′(y∣a,x), which provides samples closer to the target distribution. The gradient estimator is:

where $w'_i$wi′​ are normalized importance weights computed as $w_i = \frac{P(a|y_i,x)P(y_i|x)}{Q'(y_i|x,a)}$wi​=Q′(yi​∣x,a)P(a∣yi​,x)P(yi​∣x)​ normalized to sum to one, and $b'_i$bi′​ is a baseline derived from $Q_\theta$Qθ​ itself. The baseline subtraction exploits the identity that the expected gradient of $\log Q_\theta$logQθ​ under $Q_\theta$Qθ​ is zero, which reduces variance in the gradient estimate. This estimator corresponds to the gradient of the KL divergence between the normalized weight distributions $w'$w′ and $b'$b′, which can be monitored during training to track progress.

SteerLM 2.0 is conducted iteratively (n = 2 rounds). In each iteration, 10 diverse responses are sampled per prompt from 20,000 prompts in the Daring Anteater SFT dataset (temperature 0.7, top-p 0.9), and the model is trained for 2 epochs with the AdamW optimizer at a constant learning rate of 1e-7 and a global batch size of 128.

For inference, the authors focus on generating high-quality responses. Since HelpSteer2 responses are substantially longer and more complex than those in HelpSteer (roughly 3x), they set helpfulness, correctness, and coherence to 4, while keeping complexity and verbosity at 2, which yields better generations than setting all attributes to 4 as done in the original HelpSteer.

Experiment

Reward models trained with the SteerLM approach on the compact HelpSteer2 dataset achieved state-of-the-art RewardBench scores, with Llama 3 70B reaching 88.8% overall and Nemotron-4 340B topping the leaderboard at 92.0%, substantially outperforming models trained on larger preference datasets. Using this reward model, alignment experiments with DPO, PPO, and SteerLM on Llama 3 70B produced models that matched or exceeded Llama 3 70B Instruct, which used 100 times more data, while each method excelled on different benchmarks: DPO on TruthfulQA and Arena Hard, PPO on AlpacaEval 2.0 LC (though with degraded TruthfulQA), and SteerLM on MT-Bench. Ablations showed that reward-model-guided training consistently boosts performance regardless of SFT data quality, though better quality SFT data like Daring Anteater yields stronger overall results.

The inter-annotator agreement for HelpSteer2 attributes improved substantially after applying annotation improvements, with further gains from post-processing. Helpfulness and Correctness reached the highest agreement levels, while Coherence and Complexity remained the most challenging attributes to annotate consistently. Initial agreement for Helpfulness and Correctness was moderate but rose to around 0.79 after post-processing. Coherence and Complexity showed the lowest agreement throughout, even after improvements. Verbosity agreement improved from 0.342 to 0.548, indicating a moderate but consistent gain.

HelpSteer2 responses are rated higher across all quality attributes and are more verbose, reflecting stronger generation models. However, the correlations with helpfulness shift notably: coherence becomes less predictive, while correctness becomes a much stronger predictor, and verbosity and complexity lose influence. Coherence scores improve substantially, yet its correlation with helpfulness weakens, likely because most responses are now coherent. Correctness becomes a near-perfect predictor of helpfulness, underscoring its growing importance in assessing response quality. Verbosity and complexity show diminished correlations with helpfulness, indicating annotators focus less on style and more on factual accuracy. HelpSteer2 introduces multi-turn prompts, which have a slight negative correlation with helpfulness, suggesting follow-up responses may be harder to generate.

The table compares several proprietary reward models on Reward Bench, revealing that no single model tops every category. The Nemotron-4 340B RM leads on the primary dataset metric, while Cohere models achieve the best prior sets overall, safety, and reasoning scores. Across all models, chat and safety performance is generally high, but reasoning shows lower and more varied results. Nemotron-4 340B RM scores highest on the Reward Bench primary dataset. Cohere May 2024 outperforms others on prior sets overall and reasoning, while Cohere March 2024 posts the best safety score. GPT-4 variants trail the leading Cohere and Nemotron models on most Reward Bench metrics.

The evaluation shows that no single alignment method dominates across all benchmarks, with each technique excelling in specific areas. SteerLM achieves the highest MT Bench score, PPO performs best on AlpacaEval but degrades TruthfulQA, and Iterative DPO with Daring Anteater leads on Arena Hard and TruthfulQA. The Daring Anteater dataset generally boosts performance for SFT and iterative methods. SteerLM with DA attains the top MT Bench score, attributed to its multi-to-one contrastive learning approach. PPO yields the best AlpacaEval 2.0 LC but sharply reduces TruthfulQA performance. Iterative DPO with DA achieves the highest Arena Hard and strong TruthfulQA, indicating effectiveness on knowledge-intensive tasks.

The experiments show that annotation improvements and post-processing raised inter-annotator agreement for HelpSteer2 attributes, with helpfulness and correctness reaching the highest reliability while coherence and complexity remain hardest to label consistently. Comparing response sets, HelpSteer2 responses are rated higher across attributes but shift the correlation with helpfulness: correctness becomes a near-perfect predictor, coherence loses influence, and verbosity or complexity matter less, with multi-turn prompts adding slight difficulty. Benchmarking proprietary reward models revealed no single winner; Nemotron-4 340B RM leads on the primary dataset, while Cohere variants excel in prior sets, safety, and reasoning, and GPT-4 generally trails. For alignment methods, no technique dominates all tasks: SteerLM achieves the top MT Bench, PPO excels on AlpacaEval but hurts TruthfulQA, and Iterative DPO with Daring Anteater is best on Arena Hard and knowledge-intensive benchmarks, indicating trade-offs across evaluation dimensions.