HelpSteer2 Human Preference Alignment Dataset

* This dataset supports online use.Click here to jump.

HelpSteer2 is an open source dataset jointly created by NVIDIA and Scale AI in 2024. It aims to train a reward model that can guide large language models (LLMs) to generate high-quality answers that meet human preferences. The related paper results are "HelpSteer2: Open-source dataset for training top-performing reward models". It is an update based on the HelpSteer dataset to adapt to the current more powerful LLMs. HelpSteer2 contains about 10,000 pairs of answers. Although the number is an order of magnitude less than existing preference datasets, it is very efficient in training reward models.

The collection process of this dataset includes obtaining prompts from the ShareGPT platform and generating answers using a strong internal base model. The annotation process of the answers requires at least three annotators to annotate each answer to improve the annotation quality. Statistics from HelpSteer2 show that the model answers have higher scores in helpfulness, correctness, coherence, complexity, and verbosity compared to the HelpSteer dataset.

The HelpSteer2 dataset is very effective in training reward models. For example, the Llama 3 70B model trained with HelpSteer2 achieved a score of 92.0% on the main dataset of Reward-Bench, surpassing all public and proprietary models listed as of June 12, 2024. In addition, the research team also proposed the SteerLM 2.0 model alignment method, which can effectively utilize the rich multi-attribute scores predicted by the reward model.