One-sentence Summary

University of Washington, NYU Shanghai, and New York University propose ValueActionLens, a framework built on a 14.8k-action dataset spanning 12 cultures and 11 social topics, which reveals substantial misalignment between LLMs' stated values and their value-informed actions and demonstrates risks in relying solely on stated values to predict behavior.

Key Contributions

• The work introduces ValueActionLens, a context-aware framework for measuring value-action gaps in large language models through value-informed action generation across 132 scenarios, two evaluation tasks, and three alignment metrics.

• The work releases the VIA dataset, containing 14,784 value-informed action examples across 12 cultures and 11 social topics, to support comparison of stated values with value-informed actions.

• Experiments show substantial misalignment between generated value statements and actions across models, cultures, and social contexts, highlighting the need for context-sensitive evaluation of value-action alignment.

Introduction

As LLMs increasingly influence societal decisions, ensuring that their values and behavior align with human expectations matters for responsible deployment. Prior work has mainly assessed stated values using agreement or Likert-style responses, but it often overlooks how models act in real-world contexts and lacks a context-aware framework and dataset for systematic study. The authors leverage the value-action gap concept from psychology and introduce ValueActionLens, a framework that compares LLM value statements with value-informed actions across 132 scenarios, 12 cultures, and 11 societal topics. They also release the VIA dataset with 14,784 value-informed actions and three alignment metrics, revealing substantial context-dependent gaps between what LLMs say and how they act.

Dataset

• The authors construct a Value-Informed Actions dataset to evaluate whether language models can produce actions and explanations that align with specific human values in culturally grounded situations.

• The dataset is built from 132 scenarios formed by combining 12 countries with 11 social topics. The countries cover major English-speaking populations across regions such as North America, Europe, Australia, Asia, and Africa, while the topics come from social survey programs and include areas like social inequality, family, work, and religion.

• The value component uses 56 values from Schwartz’s Theory of Basic Values. Each value is tested under two stances, agreement and disagreement, giving 112 value stance combinations.

• Combining 132 scenarios with 112 value stance combinations yields 14,784 contextualized value-informed actions.

• Each item includes a generated action and associated explanations, with fields for the human action, attribution of value-related text spans, and a natural language explanation of the reasoning behind the action.

• The authors use a human-in-the-loop generation process. They first create 8 prompt variants for each value and scenario, then generate a 640-instance pilot set by sampling 80 actions per prompt. Two AI experts annotate the pilot data over two rounds to select the best prompt, resolving disagreements through discussion.

• Data quality is assessed with metrics for action correctness, action harmlessness, attribution sufficiency, and explanation plausibility. Prompt selection reaches substantial inter-rater reliability with Cohen’s Kappa of 0.7073.

• The final dataset is further checked through cross-cultural human evaluation. Twenty-seven annotators with relevant cultural backgrounds assess 90 randomly sampled actions and explanations, with each item reviewed by three annotators and finalized by majority voting.

• No cropping strategy is described. The main processing detail is the controlled generation and annotation pipeline used to ensure the scenarios, values, actions, and explanations are consistent and culturally contextualized.

Method

The authors leverage ValueActionLens, a framework for measuring whether value statements made by large language models are reflected in the actions they select in realistic contexts. It treats values and actions as contextual phenomena, so the design connects scenario construction, prompted evaluation tasks, and quantitative alignment scoring. Refer to the framework diagram.

At the top of the pipeline, the authors construct scenarios by combining country contexts and social topics, then pair each scenario with a structured set of universal human values. Each value is represented under both agreement and disagreement stances, producing value-context pairs for which aligned and opposing behaviors must be defined. The resulting data layer contains value-informed actions, text spans that identify action attribution, and natural language explanations that justify why an action expresses the target value.

The generation of this data is not performed by a single automatic prompt. It uses a human-in-the-loop workflow intended to improve prompt robustness, explanation quality, and cross-cultural reliability. As shown in the figure below:

Step 1 creates multiple zero-shot prompt variants for each value and scenario. Robustness is obtained by paraphrasing prompt text, reordering prompt components, and altering response requirements. Step 2 uses AI experts to inspect a subset generated from the variants and select the most effective prompt according to correctness, harmlessness, sufficiency, and plausibility. Step 3 uses the selected prompt to generate the full value-informed action set, then asks annotators from diverse cultural backgrounds to review sampled actions, explanations, and attributions, with majority voting used to finalize quality judgments.

With this data foundation, the evaluation protocol separates value declaration from value-informed behavior. Task 1 asks the model to state its inclination toward a target value in a given scenario. Prompts are structured with context, options, and requirements, and each component has two variations, yielding eight prompt variants per scenario. The context can be survey-style or portrait-style, while the response options use a Likert scale. Responses are averaged across prompt variants to produce a stable stated-value score.

Task 2 asks the model to choose between two candidate actions for the same scenario and value, one corresponding to agreement and one to disagreement. The option order is shuffled to reduce selection bias, and the same prompt variant structure is used to obtain an action-selection response. The outputs of Task 1 and Task 2 are arranged as a stated-value matrix $V$V and an action-response matrix $A$A. For scenario $i$i and value $k$k, where $K=56$K=56 values are considered, the method uses normalized response scores $v_{ik}$vik​ and $a_{ik}$aik​.

The alignment measures then compare $V$V and $A$A from three perspectives. The alignment rate converts normalized responses into binary form and computes an F1 score over stated stances and selected actions. The alignment distance preserves graded disagreement by computing element-wise $L_1$L1​ distance:

and it can be averaged over a country or topic group $C$C as:

Finally, the framework ranks values within each scenario by descending distance, identifying the largest value-action gaps for further analysis.

Experiment

The study evaluates value-action alignment in several state-of-the-art LLMs by comparing their stated values with their value-informed actions across values, countries, and social topics using multiple prompts and alignment measures. The experiments show that alignment is uneven, lower in some regions and topics, and larger for specific values such as independence and choosing one’s own goals, with misaligned values varying by cultural context. A qualitative analysis of misaligned cases further links these gaps to potential individual, interactional, and societal risks, indicating that strong benchmark performance does not guarantee consistent value-informed behavior.

The table describes a value informed action dataset built from culturally diverse scenarios spanning many countries and social topics. It covers a wide set of values and evaluates each value under both agreeing and disagreeing stances, producing thousands of actions paired with highlighted evidence and explanations. This structure supports analysis of whether model actions align with stated values in varied social contexts. The dataset covers a broad range of countries and social topics. Each value is examined under both agree and disagree inclinations. Generated actions include highlighted evidence and natural language explanations.

Human evaluation of generated value-informed actions, attributions, and explanations shows generally positive judgments from both experts and annotators. Expert ratings are consistently higher than annotator ratings across correctness, harmlessness, sufficiency, and plausibility, with plausibility receiving the strongest expert endorsement. Annotators still judge the outputs as mostly correct and sufficient, but rate harmlessness lower, suggesting that broader raters are more cautious about potentially harmful content. Experts rate the generated actions, attributions, and explanations as highly plausible. Annotators give moderately high correctness and sufficiency scores for the same outputs. Harmlessness receives the lowest relative rating from annotators compared with expert judgments.

Value-action alignment rates are generally modest and differ across models, countries, and social topics. One model achieves the strongest overall alignment, while another performs substantially worse, and some models show weaker alignment in Africa and Asia than in North America and Europe. LLMs do not consistently turn stated values into aligned actions across scenarios. Model choice strongly affects alignment quality, with one model leading and another lagging far behind.

The model comparison summarizes cross-task inconsistency, where responses shift between Agree and Disagree across two task framings. Most evaluated LLMs exhibit total misalignment in the low to mid teen percentage range, while GPT-3.5-turbo stands out with the largest share and a pronounced skew toward previously Agree responses becoming Disagree. The distribution also differs across models, indicating that inconsistency is model dependent rather than uniform. All evaluated models show cross-task inconsistency, but totals range from the low to mid teens to about one fifth of cases. GPT-3.5-turbo has the highest total misalignment and a strong skew toward the (D,A) direction. Models such as Claude-Sonnet-4, Llama-3.3-70B, and GPT-4o have more inconsistency in the (A,D) direction, showing model-specific bias patterns.

The analysis organizes misaligned value-action examples from six language models into risk categories that indicate possible harms in real-world use. Individual-level risks are most visible in the provided statistics, with discrimination and autonomy violation standing out, while interaction-level risks include overconfidence, user manipulation, and misleading explanations. Discrimination is the most frequent individual-level risk shown in the provided counts. Autonomy violation appears as a notable risk when model actions go beyond stated values. Interaction-level risks include overconfidence, user manipulation, and misleading explanations. Value-action gaps can create practical harms even when the models express socially appropriate values.

A culturally diverse value-informed action dataset evaluates whether language models translate stated values into appropriate actions under agreeing and disagreeing stances across many countries and social topics. Human evaluations suggest the generated actions, attributions, and explanations are generally plausible, correct, and sufficient, although non-expert annotators are more cautious about harmlessness than experts. Model comparisons show that value-action alignment is inconsistent across models, regions, and task framings, and that misaligned cases point to practical risks such as discrimination, autonomy violations, overconfidence, and user manipulation.