One-sentence Summary

Researchers from CISPA Helmholtz Center, TCS Research, and Microsoft demonstrate that LLM response sampling mirrors human decision-making through a dual heuristic combining a descriptive component reflecting statistical norms and a prescriptive component encoding implicit ideals, revealing that this shift toward prescriptive norms in outputs can cause significantly biased decision-making in domains such as public health and economics.

Key Contributions

• The paper develops a cognitive-science framework for LLM sampling in autonomous decision-making, proposing that sampled outputs combine a descriptive statistical component with a prescriptive ideal component, and validates the framework through controlled experiments, robustness checks, and comparisons with human studies.

• Using samples from 500 concepts across 10 domains on 15 language models, the evaluation finds statistically significant shifts from statistical norms toward prescriptive norms, and a real-world-inspired case study shows that such shifts can produce biased decision-making and ethical concerns.

• The paper further demonstrates that LLMs' prototypical representations of concepts systematically incorporate prescriptive norms, indicating that their judgments of “typical” examples are biased toward idealized versions in ways similar to human notions of normality and prototypicality.

Introduction

LLM agents often need to choose an action from many possible outputs, and exhaustive deliberation is usually impossible. In this setting, the authors study response sampling, the mechanism by which an LLM selects among candidate outputs, because this mechanism shapes downstream decisions, bias, and reliability in real applications. Prior work has shown that LLMs can form structured internal representations and behave like quick, heuristic driven agents, but it has paid less attention to what guides their probabilistic choice among options, especially when their sampled outputs do not match true statistics. The authors leverage cognitive science findings on human prototypicality to argue that LLM sampling combines a descriptive component, meaning statistical commonality, and a prescriptive component, meaning an idealized notion of what a concept should be. They support this with controlled experiments, evaluation over 500 concepts across 10 domains and 15 models, and a medical case study showing that such idealized sampling can lead to undesired outcomes.

Method

The authors leverage a cognitive theory inspired by human decision making to explain LLM sampling. Their framework treats explicit reasoning procedures such as chain of thought and other deliberative model variants as System 2 style processes, while default token generation functions as System 1 style heuristic sampling. Within this framework, sampling is described as the process by which the model probabilistically selects an output from a distribution of potential responses. The proposed theory states that LLM sampling is shaped by two latent components: a descriptive component, corresponding to the statistical norm of a concept, and a prescriptive component, corresponding to an ideal or preferred notion of that concept. Thus, the method investigates whether model outputs deviate from purely statistical regularities in a direction aligned with an internally represented ideal.

The experimental pipeline first constructs a controlled setting using a novel concept, denoted $C$C, to avoid confounding effects from concepts already embedded in the model. The descriptive norm of $C$C is instantiated with values drawn from a Gaussian distribution with mean $C_\mu$Cμ​ and known variance. The LLM is given $N$N samples from this distribution, with $N$N chosen large enough that the empirical mean closely approximates $C_\mu$Cμ​. To introduce a prescriptive norm $C_v$Cv​, each option is associated with a graded value under different experimental conditions: one in which a higher value is treated as ideal, one in which a lower value is treated as ideal, and a control condition with no explicit ideal direction. The model is then prompted to produce a sample for concept $C$C, denoted $S(C)$S(C).

To evaluate whether the sample distribution is driven only by the descriptive norm or also by the prescriptive norm, the procedure is repeated $M$M times, where $M$M is kept equal to $N$N. This yields a distribution of outputs $S(C)$S(C) for each configuration of $C_\mu$Cμ​ and $C_v$Cv​. If sampling depended solely on the statistical norm, the distribution of $S(C)$S(C) should resemble the input distribution. To separate genuine prescriptive deviation from a failure to estimate input statistics, the model is also asked to report the average of the input distribution, denoted $A(C)$A(C). When $A(C)$A(C) closely matches $C_\mu$Cμ​, the authors interpret the model as successfully approximating the descriptive norm. The deviation of $S(C)$S(C) from $A(C)$A(C) is then tested using the Mann Whitney U test, with $p < 0.05$p<0.05 treated as evidence of a statistically significant shift.

For existing concepts, the method is extended to settings where $C_\mu$Cμ​ and $C_v$Cv​ are not externally known. The pipeline instead elicits a self reported average $A(C)$A(C), a self reported ideal value $I(C)$I(C), and a sampled value $S(C)$S(C). Since the ideal may lie above or below the average, the authors define a directional deviation metric $\alpha$α. For a sample $S(C)$S(C) of concept $C$C, the metric is computed as

A positive value of $\alpha$α indicates that the model’s sample moves from the average toward the ideal. Because raw $\alpha$α depends on the numerical scale of the concept, the method also computes a normalized form,

This normalization places $A(C)$A(C) at the origin and $I(C)$I(C) at unit distance, making the deviation comparable across concepts with different value ranges. Degenerate cases where $A(C) = I(C)$A(C)=I(C) are excluded because no meaningful direction toward the ideal can be defined.

As shown in the figure below:

The visualization illustrates how average, ideal, and sample values are interpreted within the framework. When the ideal lies below the average, a positive $\alpha$α is produced if the sample falls below the average and closer to the ideal. When the ideal lies above the average, a positive $\alpha$α is produced if the sample rises above the average and moves toward the ideal. The normalized value $\hat{\alpha}$α^ further indicates whether the sample exceeds the ideal, equals the ideal, lies between the average and ideal, equals the average, or falls on the side opposite the ideal. Across many concepts, the authors compare the number of samples falling on the ideal side of the average against a binomial null hypothesis in which the probability of ideal side placement is $0.5$0.5. This provides a statistical procedure for measuring whether prescriptive norms systematically influence LLM sampling.

Experiment

The experiments evaluate whether LLM sampling is shaped not only by statistical averages but also by prescriptive ideals, using both controlled novel concepts with assigned value systems and real-world concepts across multiple domains and models. The results show that when no clear ideal is present, samples largely follow the statistical distribution, whereas introducing a positive or negative value system shifts samples toward the ideal side of the average, and this pattern also appears in judgments about existing concepts and prototypes. Comparisons with human studies suggest a broadly similar influence of prescriptive norms, although LLMs often exhibit stronger or more absolute idealization, indicating potential fairness and alignment concerns.

Across settings, the reported average stays close to the given distribution mean, while the mean of sampled values tracks the assigned prescriptive norm. Under neutral control, sampled values do not differ significantly from the reported average or the true mean. When a positive norm is emphasized, samples shift upward, and when a negative norm is emphasized, samples shift downward, with the effect visible for both unimodal and bimodal inputs. Reported averages align with the true distribution mean across all conditions Sample means shift toward positive norms and away from negative norms Neutral controls show sampling that matches the statistical average

Across the listed language models, sampled concept values tend to shift away from descriptive averages and toward idealized values, indicating a prescriptive component in model judgments. This tendency is generally stronger in larger models and in chat or instruct variants compared with pretrained base models. The results suggest that prescriptive influence is present after pretraining and is amplified by alignment training. Larger models tend to show stronger movement toward ideal values. Chat or instruct variants generally show greater prescriptive influence than base models. The deviation is statistically significant for most models, supporting a systematic bias.

The averaged concept scores show that prototypical judgments do not collapse to simple averages; across the tested concepts, prototype ratings are elevated relative to average ratings. They also frequently align more closely with ideal ratings, indicating that prescriptive norms shape how typical examples are represented. This pattern is strongest for concepts such as dog, grandmother, hospital, and stereo speakers, where the prototype sits nearer the ideal than the average. Across the tested concepts, prototypical scores exceed average scores, suggesting judgments favor idealized over merely descriptive representations. Prototypical scores often track ideal scores more closely than average scores, with stereo speakers and vacation showing prototype ratings between the two rather than simply matching either. Salad shows an outlier pattern in which the prototype rating exceeds both the average and ideal ratings, indicating the ideal component does not fully determine prototypicality.

Across controlled distribution tasks and model comparisons, the experiments examine whether reported averages, sampled values, prototype ratings, and ideal or prescriptive norms diverge. They find that reported averages track true statistical means, while sampled values and prototype judgments are systematically shifted toward assigned or idealized norms, with stronger effects under positive or negative norm cues. This prescriptive bias appears across language models, especially larger and chat or instruct variants, suggesting alignment amplifies idealized norms, and it also shapes prototypicality judgments for everyday concepts, although some concepts show idiosyncratic deviations.