One-sentence Summary

University of Oxford, University of Pennsylvania, Bocconi University, et al. introduce PRISM, a dataset mapping the sociodemographics and stated preferences of 1,500 diverse participants from 75 countries to fine-grained feedback across 8,011 live conversations with 21 LLMs, enabling participatory, representative, and individualized alignment data for studying subjective and multicultural disagreements.

Key Contributions

• PRISM is a dataset linking the sociodemographics and stated preferences of 1,500 participants from 75 countries to contextual preferences and fine-grained feedback from 8,011 live conversations with 21 LLMs.
• The dataset expands geographic and demographic participation, includes census-representative samples for the UK and US, and provides individualized ratings linked to detailed participant profiles for personalization and attribution of sample artefacts.
• The paper uses PRISM in three case studies on subjective and multicultural perspectives in value-laden and controversial issues, demonstrating the need to account for which humans provide which alignment data.

Introduction

As large language models are aligned through human feedback, the design of feedback data directly shapes which preferences and values systems learn. Existing feedback datasets often rely on binary comparisons, narrow annotator pools, limited participant information, and weak documentation of values, which can collapse personal and cultural context into reductionist labels. The authors introduce PRISM, a human feedback dataset that links detailed survey responses, participant profiles, stated preferences, and live LLM conversations from 1,500 English speaking participants, including census representative UK and US samples. This resource supports personalised alignment, value consensus analysis, welfare studies, and research into how sampling choices and disagreement affect model behaviour.

Dataset

• The authors describe PRISM, a human preference and metadata dataset for LLM alignment built from 1,500 English-speaking participants recruited through Prolific. Data collection ran from 22 November to 22 December 2023 and used informed consent, pseudonymised identifiers, and PII checks. The dataset is deliberately diverse but still skews White, Western, educated, and English-speaking.

• The data are built from two linked stages. First, each participant completes a survey that creates a profile with standard demographics, LLM familiarity and usage, a self-written “constitution” or system string, an open-ended values description, and structured stated preferences for LLM behaviour. Demographic coverage includes people born in 75 countries, and 84% of participants report direct or indirect LLM use.

• The survey stage uses filtering and quality controls: participants must be over 18, give informed consent, read a brief LLM primer, and are discouraged from submitting LLM-written answers. Stated preferences use 0 to 100 visual analogue scale ratings, with numeric values hidden from participants. Open-ended self-described ethnicity and religion are manually annotated into broader categories for analysis.

• The conversation stage collects in-situ preference data. Participants complete two conversations for each of three prompt types, unguided, values guided, and controversy guided, for six conversations in total. For each conversation, participants write an opening prompt and receive up to four responses from different LLMs, rating each on a hidden 1 to 100 scale.

• Within each conversation, the highest-scoring LLM from the opening turn is locked for later turns. Participants must continue for at least one more turn, with overall conversation lengths varying from 2 to 10 turns and a mean of 3.4 turns. In later turns, participants rate two non-deterministic responses generated by the selected model.

• Conversation metadata includes model identity, response ratings, prompt type, conversation length, and fine-grained post-conversation feedback. Participants rate performance statements, choice statements, and give open-ended conversation feedback, averaging 29 words. The same behaviour attributes are reused in the survey and conversation stages to compare stated preferences with contextual preferences.

• Model responses are sourced from 21 LLMs, including 9 open-access and 12 commercial API models. To reduce length confounds and fatigue, system prompts instruct models to keep responses to 50 words or fewer. The dataset is reported as relatively low toxicity at 0.06%, and it is not moderated before release so it can support conversational safety research.

• The paper uses PRISM as a preference feedback dataset rather than only as a generic prompt-response corpus. The 1 to 100 cardinal ratings can serve as preference intensity scores or be converted into rankings for reward-modelling style analyses. Participant profiles are paired with in-situ ratings to study how individual characteristics relate to real-time model preferences.

• The excerpt does not specify a formal training split, mixture ratio, or cropping strategy. The main processing details are survey-profile construction, demographic annotation, PII handling, hidden numeric ratings, fixed top-model selection for later turns, length-control system prompts, and the collection of open-ended feedback as conversation metadata.

Method

The method builds an alignment dataset through a staged human interaction pipeline rather than a single preference labeling task. The process begins with a participant survey that establishes a profile and stated preferences before any model conversation occurs. The survey collects information about prior exposure to and use of large language models, and it also asks participants to write a self defined system string, described as a constitution or set of governing principles for how a model should behave. This survey stage creates the context for later conversational ratings by capturing both stated preferences and individual values.

After the survey, participants enter real time conversations with models through a custom interface built on the Dynabench platform. To encourage diversity across the objective and subjective spectrum, participants complete two conversations in each of three conversation types, producing six conversations per participant. These conditions are unguided, values guided, and controversy guided. The unguided condition allows free choice of topic, while the values guided and controversy guided conditions prompt participants to focus on personally meaningful issues or socially contested topics. This design is intended to elicit a wider range of human values and preference signals than would arise from unconstrained prompts alone.

The first turn of each conversation is a multi model comparison step. Participants write a free text opening prompt and receive up to four responses from different large language models. Each response is rated using a visual analogue scale anchored from terrible to perfect. The slider position is recorded as a numeric score from 1 to 100, but the number is hidden from the participant to reduce anchoring and dependence between conversation ratings. The authors choose this cardinal feedback because it encourages subjectivity, allows downstream conversion to rankings if desired, and captures preference intensity beyond binary chosen and rejected comparisons. The highest scoring model from the first turn is then locked in for subsequent turns, with random tie breaking when scores are identical.

Subsequent conversation turns focus on within model comparison. Participants continue the interaction for at least one more turn, with an encouraged range of two to ten total turns to avoid introducing an artificial length bias. The mean conversation length is $\mu_T = 3.4$μT​=3.4 with standard deviation $\sigma_T = 1.6$σT​=1.6. In later turns, participants rate two responses generated from the same selected model using non deterministic decoding. Because both responses come from the same model, they are more similar in style and content than the first turn responses, and score differences are typically narrower. This creates a complementary signal to the cross model comparison in the opening turn.

After the conversation ends, the pipeline collects contextual fine grained feedback. Participants first evaluate the performance of their highest rated model from the first turn against attributes such as fluency, factuality, safety, diversity, creativity, helpfulness, and value alignment. They then rate how important those attributes were for their choice among models. The same attributes also appear in the initial survey as stated preferences, allowing comparison between abstract values and context dependent judgments. As shown in the figure below, the attribute schema links stated preferences in the survey to performance attributes and choice attributes collected after each conversation.

This three part attribute design makes the pipeline useful for reward modeling and preference analysis. Stated preferences capture broad attitudes before the interaction, performance attributes capture how well a selected model fulfilled specific qualities, and choice attributes capture why that model was preferred relative to alternatives. The final stage also collects open ended natural language feedback about the conversation as a whole, including what was good, what was bad, what was missing, and how the interaction could have been improved.

The system design also incorporates sampling and model diversity to reduce bias in the resulting training signal. Participant recruitment is organized around two complementary paths: census representative samples where census matched quotas are available, and balanced global samples where country specific studies allocate quota by global region to prevent overrepresentation of particularly active participant pools. Gender balance is targeted within national samples when possible. For model selection, the pipeline uses a model agnostic setup with a set of 21 large language models spanning open access and commercial API families, varying parameter sizes, and different levels of existing safety or alignment behavior. To control for response length and participant fatigue, system prompts ask models to keep responses no longer than 50 words. This controlled yet diverse generation environment produces preference data that reflects both broad demographic contexts and varied model behaviors.

Experiment

Experiments using the PRISM dataset evaluate how identity and context shape people's initial conversations with LLMs, model preferences, and the welfare impact of selecting models from sampled feedback. Topic clustering and regression show that demographic groups overdiscuss distinct issues, yet diverse users also share common semantic spaces, while pairwise preference aggregation reveals that model rankings vary with conversation topic and background, and some style features explain only part of scoring differences. Welfare simulations further show that smaller or biased samples can degrade outcomes for underrepresented groups and that no single model satisfies all users.