One-sentence Summary

Researchers from the University of Toronto, Microsoft Research, Johns Hopkins University, and MILA propose a two-step context-aware preference modeling approach that first selects a context to resolve under-specification and then evaluates preference relative to that context, decompose reward modeling error according to these two steps to justify supervising context alongside preference, release context-conditioned preference datasets showing that existing models benefit from but fail to fully consider added context, and demonstrate that their finetuned context-aware reward model achieves context-specific performance exceeding GPT-4 and Llama 3 70B on the tested datasets.

Key Contributions

• The paper introduces the RPR datasets, the first open-source context-conditioned preference datasets in which preferences are strongly determined by context, addressing the underspecification of direct preference feedback in language model alignment.
• Experiments show that existing preference models benefit from added context but frequently fail to fully consider it, sometimes ignoring context entirely, revealing a critical gap in context-specific preference evaluation.
• Finetuning a context-aware reward model achieves context-specific performance exceeding that of GPT-4 and Llama 3 70B on the tested datasets, and a single persistent context, potentially inferred from data, may be sufficient to substantially improve preference prediction.

Introduction

As language models and foundation models expand into real-world applications that require following diverse instructions and executing long-range tasks, ensuring their behavior matches human expectations becomes critical. The standard approach aligns LMs to human preferences by finetuning on pairwise preference comparisons between outputs. This method struggles with ambiguous preference queries, since user identity, goals, and other contextual factors are often unspecified. Human annotators agree only about 65 percent of the time on binary preference judgments, and when models are trained on such non-basic preferences, implicit contextual biases get embedded into the model. Prior work using the Bradley-Terry model implicitly aggregates preferences across hidden contexts with the Borda rule, which can under-serve certain subgroups.

The authors propose a two-step, context-aware preference modeling framework. The first step resolves underspecification by selecting a context, and the second step evaluates preference relative to that chosen context. This decomposition explicitly surfaces the contextual assumptions underlying preference, shifts alignment from pure preference modeling to a hybrid of preference modeling and context supervision, and enables pluralistic adaptation to diverse users. The authors prove an upper bound on reward modeling error as the sum of context inference error and context-specific reward modeling error, which justifies supervising context as a viable alignment path. To support this, they introduce open-source context-conditioned preference reversal (RPR) datasets where preference for a given prompt flips under an alternative context. Their experiments show that current models benefit from added context but sometimes fail to fully weigh it, and that finetuning on RPR data substantially improves context-specific preference prediction. They also demonstrate that a single persistent context, such as a user profile or system prompt, can be inferred from as few as 16 to 32 expressed preference samples and applied across many queries.

Dataset

The authors introduce the Reasonable Preference Reversal (RPR) datasets, a set of synthetic preference datasets designed to measure whether preference prediction is sensitive to context. The datasets include over 20,000 paired tuples of prompt, context, and preference judgments, formatted as $(x, z, y_1 \succ y_2)$(x,z,y1​≻y2​). The samples are constructed so that preference between two completions for the same prompt is entirely ambiguous without context; for every context, there is an alternative context for which the preference reverses. This design ensures that performance on the dataset is determined solely by the model's ability to interpret the context.

The dataset comes in two base versions:

• RPR Criteria: Context is a short sentence describing which features are important, such as "Uses a formal and professional tone that emphasizes the technical aspects of C++".
• RPR Scenarios: Context describes the scenario that led to the underspecified prompt, such as a marketing manager preparing for a product launch.

The data was generated by GPT-4 Turbo using a series of prompts designed to maximize validity and avoid invalid or ambiguous samples. Small-scale human validation through blind preference queries found agreement rates of at least 95% on both splits.

The dataset is divided into a training set of 10,167 paired samples and a test set of 1,000 paired samples, with no overlap between train and test prompts. The paper also extends the dataset with user profiles in a separate section. The datasets are publicly available at https://huggingface.co/datasets/microsoft/rpr.

Method

The authors introduce an intent-utility formalism to model user-LM interactions, defined by the tuple $(\mathcal{T}, \mathcal{X}, \mathcal{Y}, u)$(T,X,Y,u), where $\mathcal{T}$T denotes the space of intents, $\mathcal{X}$X is the space of prompts, $\mathcal{Y}$Y is the space of completions, and $u: \mathcal{T} \times \mathcal{Y} \to \mathbb{R}$u:T×Y→R is a scalar utility function. Following the standard Bradley-Terry model, the probability of preferring completion $y_1$y1​ over $y_2$y2​ given a specific intent $i$i is expressed as:

where $\sigma$σ is the logistic function. A critical distinction in this formalism is that preference and utility are defined with respect to intents rather than prompts. A user with intent $i$i selects a prompt $x \in \mathcal{X}$x∈X to query the model, and an annotator presented with a preference query $(x, y_1, y_2)$(x,y1​,y2​) implicitly infers intent $i$i from $x$x, then samples a preference from the Bernoulli distribution $B[p(y_1 \succ y_2 \mid i)]$B[p(y1​≻y2​∣i)].

Both users and annotators may hold a distribution over intents rather than a single fixed intent. The authors assume a base distribution $\bar{p}(i)$pˉ​(i) over possible intents, along with a conditional distribution $p(x \mid i)$p(x∣i) over prompts given intents, which induces a natural inference distribution $p(i \mid x)$p(i∣x) for any prompt. Under this view, the prompt is a partial specification of intent; additional context $z \in \mathcal{Z}$z∈Z, where $\mathcal{Z}$Z forms a discrete partition of the intent space, can be appended to obtain an extended prompt $(x, z)$(x,z). To measure utility under a distribution $p$p of intents, the authors adopt an expected utility model:

Although standard RLHF does not align with this model, it satisfies desirable axioms for "ideal" preference annotation. Defining $u(x, y) := u(p(\cdot \mid x), y)$u(x,y):=u(p(⋅∣x),y), linearity of expectation yields the decomposition $u(x, y) = \sum_{z \in \mathcal{Z}} p(z \mid x) u((x, z), y)$u(x,y)=∑z∈Z​p(z∣x)u((x,z),y). For convenience, the utility difference $\Delta(\cdot, y_1, y_2) := u(\cdot, y_1) - u(\cdot, y_2)$Δ(⋅,y1​,y2​):=u(⋅,y1​)−u(⋅,y2​) also decomposes linearly:

In standard RLHF, a dataset of prompt-preference tuples $\mathcal{D} = \{(x, y_1 \succ y_2)\}$D={(x,y1​≻y2​)} is used to learn a utility estimator $\hat{u}: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$u^:X×Y→R, conventionally referred to as the reward model. The authors posit the existence of a context inference model $\hat{p}(z \mid x)$p^​(z∣x) that satisfies the analogous relation:

While standard RLHF never learns such an estimator, the analysis assumes it exists either implicitly or explicitly, and that $\hat{u}(x, y)$u^(x,y) is computed via the decomposition rather than by direct evaluation. Since $\hat{u}$u^ is only unique up to constant shifts, accuracy is measured via the utility difference estimator $\hat{\Delta}(x, y_1, y_2) := \hat{u}(x, y_1) - \hat{u}(x, y_2)$Δ^(x,y1​,y2​):=u^(x,y1​)−u^(x,y2​).

The absolute error for a particular preference query can be bounded as follows. Letting $\Delta_z$Δz​ denote $\Delta((x, z), y_1, y_2)$Δ((x,z),y1​,y2​), the authors derive:

The equality follows from adding and subtracting $\sum_{z} p(z \mid x) \hat{\Delta}_z$∑z​p(z∣x)Δ^z​ and rearranging, and the final inequality applies the triangle inequality multiple times. In the case where a single context is assumed (i.e., $p(z = c) := 1$p(z=c):=1) and a single prediction is made ($\hat{p}(\hat{c}) = 1$p^​(c^)=1), the bound simplifies to:

Both the general and specific bounds formalize a central claim: if predictions can be made accurately given the true context (or context distribution), then the preference modeling problem reduces to a context inference problem.

The upper bound decomposes total error into a context-weighted prediction error and a preference-weighted inference error. At one extreme, setting $\mathcal{Z} = \emptyset$Z=∅ recovers standard preference modeling, where context inference error is zero and all error stems from preference prediction given the prompt alone. At the other extreme, letting $\mathcal{Z}$Z be degenerate such that $z$z directly encodes the preferred response (e.g., "the preferred response is $y$y") makes the prediction error vanish, but the context inference problem becomes equivalent to generation. Between these extremes, the authors conjecture a smooth trade-off between prediction error and inference error, which would hold if a single context can disambiguate a range of different preference queries. Their experiments support this conjecture: conditioning on specific criteria outperforms conditioning on more abstract scenarios, which in turn outperforms conditioning on a single user profile applied to all queries at once.

When the utility estimator $\hat{u}$u^ is sufficiently accurate given additional context, the preference modeling problem is largely reducible to context specification. In this regime, the authors suggest allocating annotation resources toward specifying additional context rather than having annotators rank completions for ambiguous prompts. These annotations can then train a context inference model that disambiguates prompts. The authors hypothesize that the cardinality of the context space is smaller than that of the completion space given a prompt, making joint context-preference annotation more data-efficient than preference annotation alone. The effectiveness of this approach hinges on whether language models are genuinely sensitive to added context, a question the authors evaluate on real data and address by training context-conditioned reward models.

Experiment

The experiments evaluated context-aware preference modeling across multiple datasets and models, including reward models, generative judges, and a finetuned Mistral CARM, using agreement metrics with context-conditioned preferences. Results showed that access to context generally improves performance, though models sometimes ignore adversarial contexts, revealing room for improvement. Finetuning on the RPR datasets markedly enhanced context-specific performance, matching or exceeding larger models like GPT-4 Turbo, and the models generalized to augmented real-world preference datasets. Finally, a single inferred user profile could recover about 80% of preference agreement, with only 2-4 samples resolving default biases and 16-32 samples recovering most of the benefit of ground truth context.

Context-aware preference modeling addresses ambiguity by making implicit assumptions explicit, improves adaptability to new users and use cases, and enables flexible aggregation via methods like jury learning and expected utility. Finetuning a context-aware model also yields marked improvements in context-specific performance, often matching or exceeding larger baseline models. Making context explicit resolves a key source of ambiguity in preference queries, shifting alignment burden to fine-grained context supervision. Incorporating context allows models to better steer toward new users and use cases. Aggregation flexibility expands beyond the Borda rule to include jury learning and expected utility. A finetuned context-aware model shows better context-specific performance than its base model, rivaling or surpassing larger models like GPT-4 Turbo and Llama 3.

Adding context generally improves preference evaluation across datasets, but the benefit is most pronounced when preferences are ambiguous without context. A model finetuned on the RPR training sets substantially outperforms its unfinetuned counterpart on test sets. Larger and instruction-tuned models tend to gain more from added context. Access to context improves performance across all tested datasets, with the greatest gains on datasets where preferences are ambiguous without context. Finetuning on RPR training sets yields substantially better test performance compared to the unfinetuned model. Larger and instruction-tuned models benefit more from added context.

Current models are largely insensitive to strongly phrased adversarial context, often failing to adjust their predictions even when it should reverse or randomize them. The finetuned CARM model outperforms other small models on both adversarial criteria, yet still falls short of optimal behavior, indicating substantial room for improvement. Even with strongly adversarial context, many models retain near-baseline scores, showing they often ignore added instructions. The finetuned CARM achieves the best performance among small models on both nonsense and negative criteria. Performance across all models remains far from optimal, especially on the nonsense criterion where random scoring would yield 0.50.

Adding context to preference datasets generally improves model performance across all tested models, but the finetuned context-aware reward model (CARM) stands out among small models when given high-quality or privileged context. CARM also shows consistent gains over its base model in most context-augmented settings, matching or exceeding the performance of larger models. The finetuned CARM outperforms other small models on context-augmented preference tasks, especially when the context includes privileged preference information. Context generally boosts performance for most models, but CARM's improvements are more pronounced, particularly on the HHH and Reward Bench datasets with context. CARM's context-specific gains over its base model are largest when the context is inferred from both the prompt and the expressed preference, indicating its strength in leveraging rich contextual signals.

Experiments with five synthetic user profiles show that a single context can effectively disambiguate preferences across many prompts. The finetuned CARM model and Llama 3 Instruct achieved the highest agreement with GPT-4's profile-conditioned labels, surpassing GPT-4 itself. This suggests that context-aware finetuning can improve preference modeling even when the context is a broad user profile. Finetuned CARM and Llama 3 Instruct outperformed GPT-4 in recovering profile-conditioned preferences. Profile-conditioned performance varied strongly across profiles, with the best models reaching about 80% agreement. Smaller finetuned models matched or exceeded larger models, indicating that targeted finetuning can offset model size advantages.

Across experiments, incorporating explicit context into preference modeling consistently improves evaluation accuracy, particularly for ambiguous preferences, with finetuned context-aware reward models (CARM) often matching or exceeding larger baselines like GPT-4 and Llama 3. However, models remain largely insensitive to strongly adversarial context, and performance still falls short of optimal behavior, indicating room for improvement. In synthetic user profile tests, a single context effectively disambiguates preferences, with finetuned CARM and Llama 3 Instruct outperforming GPT-4 in recovering profile-conditioned preferences, showing that targeted finetuning can offset model size advantages.