Difference Aware Fairness Difference Perception Benchmark Dataset

*This dataset supports online use.Click here to jump.

Difference Aware Fairness is a difference perception benchmark dataset released by Stanford University in 2025. The related paper results were published in ACL 2025:Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMs" and was awarded the best paper, which aims to measure the performance of the model in difference perception and context awareness.

The dataset contains eight benchmarks, divided into two types: descriptive and normative tasks, covering a variety of real-world scenarios, including legal, professional, and cultural fields. Each benchmark contains 2,000 questions, 1,000 of which require distinguishing between different groups, for a total of 16,000 questions.

Descriptive Tasks

• Religion: Based on the religious population ratio data of different countries, determine which countries have a higher proportion of specific religious groups.
• Occupation: Assess differences in representation of different genders, races, and ethnicities in specific occupations, based on data from the U.S. Bureau of Labor Statistics.
• Legal: This refers to special treatment based on group differences permitted under U.S. law.
• Asylum: This policy determines which religious minorities have grounds to apply for asylum in the United States based on the discrimination they face from governments and society in various countries.

Normative Tasks

• Bias Benchmark for QA (BBQ): Based on the BBQ dataset, it evaluates the model's ability to identify harmful assumptions.
• Social Bias Frames: Comparing the relative harms suffered by different groups in a specific context.
• Occupation Affirmative Action: Explores whether affirmative action is needed for disadvantaged groups in specific occupations to correct historical discrimination.
• Cultural Appropriation: Determine which groups are more appropriate to participate in certain cultural activities in a specific context to avoid the harm caused by cultural appropriation.