The RPR (Reasonable Preference Reversal) dataset is a synthetic conditional-preference dataset released by Microsoft Research in 2024, associated with the paper "Improving Context-Aware Preference Modeling for Language Models", aiming to advance research and development in natural language processing (NLP) tasks, particularly in the development of context-aware preference and reward models.

The dataset contains over 20,000 paired tuples, each consisting of a prompt, context (criteria or scenario), and preference judgments. Samples appear in pairs, ensuring that without context, the preference between two completions for the same prompt is completely ambiguous, thereby ensuring that preference prediction performance depends solely on the model’s ability to understand and interpret context.

Dataset Composition

The dataset consists of two splits: a training set and a test set. The training set contains 10,167 samples, and the test set contains 1,000 samples. Each sample includes the following fields:

• prompt: The question or prompt content posed by the user.
• response_a, response_b: The two responses to be compared.
• criteria_x, criteria_y: Different evaluation criteria used to judge response preferences.
• category_x, category_y: Category information corresponding to the evaluation criteria.
• scenario_x, scenario_y: Different application scenarios used to judge response preferences.
• profile_0 to profile_4: User profile information relevant to preference judgment.
• id: The unique identifier for the sample.

Citation

@misc{pitis2024improvingcontextawarepreferencemodeling,
      title={Improving Context-Aware Preference Modeling for Language Models}, 
      author={Silviu Pitis and Ziang Xiao and Nicolas Le Roux and Alessandro Sordoni},
      year={2024},
      eprint={2407.14916},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.14916}, 
}