One-sentence Summary

Amazon and Aequa-Tech introduce MultiPICo, a multilingual perspectivist corpus of ironic short conversations from Twitter and Reddit with disaggregated annotations and sociodemographic metadata, demonstrating that cultural factors influence irony perception and enabling benchmarks of large language models' irony recognition, positionality with respect to demographic groups, and efficacy of perspective-taking prompting.

Key Contributions

• Introduces MultiPICo, a multilingual perspectivist corpus of ironic short conversations from Twitter and Reddit, covering 9 languages and 25 high- and low-resourced linguistic varieties, with Post-Reply pairs, disaggregated irony annotations, and sociodemographic metadata for annotators.

• Analyzes irony annotation disagreement across sociodemographic groups, revealing high variability in perception, strong polarization linked to age and student status, and in German, English, and Spanish, a connection between annotator nationality and agreement across linguistic varieties.

• Benchmarks large language models for irony detection in perspectivist settings by comparing prompts without sociodemographic information, prompts with trait-based perspectives, and prompts with data-driven perspective examples, while positionality analysis shows ChatGPT aligns more closely with the youngest annotators.

Introduction

The authors leverage AI’s growing role in understanding social media communication, where irony detection supports moderation, inclusivity, and evaluation of large language models, while showing that dataset design and annotator demographics shape model behavior. Prior work often treats subjective language judgments as single labels, and existing disaggregated irony datasets are scarce, narrow, and mostly English, leaving multilingual variation and sociodemographic disagreement poorly covered. Their main contribution is MultiPICo, a multilingual perspectivist corpus of Post-Reply pairs in nine languages and 25 varieties, released with disaggregated irony annotations and annotator metadata, used to analyze how demographics affect irony perception and to benchmark LLM positionality with sociodemographic prompting.

Dataset

• The authors use MultiPICo, a corpus of 18,778 short Post and Reply conversations collected from Reddit, 8,956 pairs, and Twitter, 9,822 pairs, across 9 languages and 25 linguistic varieties.

• The Reddit portion was collected from January 2020 to June 2021 using the Pushshift repository. The authors selected 26 subreddits and inferred linguistic variety from subreddit membership. They kept conversation-starting messages and direct replies, mostly first-level comments with some second-level comments.

• The Twitter portion was collected through the Twitter Stream API. The authors used geolocation, excluded quotes and retweets, retrieved full conversations, and retained tweets that directly replied to the starting post.

• The authors applied filtering rules to remove pairs containing deleted or removed comments and to verify target languages with the language identification library LangID. They tried to balance Reddit and Twitter pair counts, but Reddit lacked enough data for US-American Spanish, Swiss-German, Iraqi Arabic, and Yemeni Arabic.

• Each example is a Post and Reply pair with associated metadata, including original Post and Reply IDs, subreddit names, and geolocation information. The final annotation metadata also records annotator gender, nationality, age group, ethnicity, student status, and employment status.

• The authors collected irony labels on the crowdsourcing platform Prolific using a custom interface that presented the Post as context and asked whether the Reply was Ironic or Not ironic. Each instance received a mean of 5.02 annotations, with limits on how many times one instance could be annotated.

• Annotators had to meet a minimum 99% completion rate, be native speakers of the target language, and be balanced across gender and nationality where possible. The authors hired about 24 annotators per linguistic variety, except English, where they hired about 15, and could not preserve balance for Arabic due to annotator scarcity.

• Annotation quality was checked with embedded attention questions at a 1% probability. Annotators with fewer than 50% correct attention checks were excluded, leaving 506 annotators in the final annotation pool.

• The paper uses MultiPICo as a human-annotated irony corpus and as the basis for exploratory analysis of annotation polarization across sociodemographic dimensions. The provided text does not describe a model training split, mixture ratio, or cropping strategy.

Method

The authors design MultiPICo as a short-conversation corpus and annotation pipeline rather than as a model architecture. The central goal is to construct context-aware Post–Reply pairs and collect human judgments on whether the Reply contains irony. The pipeline combines source-specific data harvesting, linguistic and quality filtering, interface-based annotation, annotator filtering, and demographic collection.

The corpus is built from two social media sources: Reddit and Twitter. For Reddit, the authors harvest comment threads and retain the first comment in most cases, with a small number of second-level comments included when appropriate. The Reddit data are collected through Pushshift over a fixed time window from January 2020 to June 2021. To cover multiple linguistic varieties, the authors select subreddits associated with particular language communities and infer the variety of Reddit users from the subreddit context. During filtering, Post–Reply pairs containing deleted or removed content are excluded. Target-language identification is further performed using LangID.

For Twitter, the authors collect data through the Twitter Stream API and use geolocation information to support language-variety coverage. Quotes and retweets are excluded, and the authors retrieve the full conversation around a selected tweet. They then retain tweets that directly reply to the original starting message. The intended design is to obtain a comparable number of Post–Reply pairs from Reddit and Twitter, although some linguistic varieties have insufficient Reddit data, which leads to uneven source coverage across varieties.

The annotation process is implemented on Prolific through a custom-built interface. The interface presents the Post as conversational context and asks the annotator to classify the Reply as ironic or not ironic. This design reduces the annotation task to a binary decision while preserving the short conversational structure needed for irony interpretation. The authors also store metadata for each instance, including Post and Reply identifiers, subreddit information for Reddit data, and geolocation data for Twitter data.

To ensure reliable annotation, the authors enforce a per-instance annotation budget with a mean of approximately $5.02$5.02 annotations per instance. Annotator selection is based on strict criteria. Each annotator must have a completion rate of at least $99\%$99% and must be a native speaker of the target language. The recruited annotator pool is additionally balanced across gender and nationality, with roughly $24$24 annotators for each linguistic variety in most cases and about $15$15 annotators for the English subset. For some Arabic varieties, the authors report that balance could not be preserved because of limited annotator availability.

Quality control is integrated into the annotation task itself. The authors insert attention-check questions with a probability of $1\%$1%, formatted as explicit instruction-following prompts such as requiring the annotator to answer a specified value. Annotators must achieve at least $50\%$50% correct responses on these checks. Annotations from annotators who fail the threshold are removed, resulting in a final pool of $506$506 annotators. In addition to binary irony labels, the authors collect demographic attributes such as age group, ethnicity, student status, and employment status to support downstream analysis of annotator variation.

Experiment

The experiments use polarization analysis of MultiPICo to compare instance-level divergence across sociodemographic groups and nationality-based splits with randomized baselines, showing that irony interpretation is generally language-shared but becomes culturally sensitive for specific linguistic varieties. They then benchmark zero-shot LLMs on aggregated and perspectivist evaluations, revealing that current models struggle to detect irony, especially in open and non-mainstream language settings. Additional analyses of model positionality and perspective-taking prompting show that LLMs may inherit annotator biases and that data-driven few-shot examples can improve perspective-sensitive irony detection more reliably than fixed demographic prompts across some languages.

The table summarizes multilingual annotation coverage for short Post and Reply conversations, showing variation in annotator pools, annotation totals, text counts, and source composition across languages. Most shown languages have annotation means close to five per text, while label rates consistently favor not ironic labels over ironic labels. Hindi shows the highest annotation mean and the most balanced ironic versus not ironic distribution among the listed languages. Language coverage is uneven, with some languages showing much larger text and annotation volumes than others. Reddit and Twitter contribute to the shown language splits, but their relative shares vary by language.

Across languages, observed annotator agreement usually exceeds random agreement for most demographic dimensions, with only isolated negative gaps. The strongest relative polarization appears for generation and student status, indicating that these traits most clearly separate annotators in irony judgments. Ethnicity also shows notable polarization in several languages, while some comparisons are missing or weakly defined. Most real P-index values are higher than their random baselines. Generation and student status are the most consistently polarizing dimensions. A few language and dimension combinations show near-zero or negative differences. Ethnicity produces especially large relative gaps in some languages.

The examples illustrate how irony annotations can be divided by sociodemographic groups such as gender. In one example, female and male annotators share the same interpretation, leading to no cross-group polarization. In another, each group agrees internally but assigns different labels to the same post-reply pair, indicating strong polarization and perceived ambiguity. Internal agreement within gender groups can coexist with disagreement between groups High polarization flags instances that may benefit from perspective-taking prompting Gender-based annotation groups reveal ambiguous irony examples

The P-index trends suggest that irony interpretation is often country-based, with stronger agreement when annotators judge a variety of their own language and culture. This pattern is clearest for German and English varieties, and it is also observed for Mexican Spanish. French and Portuguese show weaker or less specific alignment, with certain varieties driving polarization more than annotator origin. German and Austrian irony shows the clearest nationality-language match. Indian English produces stronger agreement among Indian annotators, with similar corresponding-country patterns for American and British English. Mexican annotators report higher irony agreement on Mexican Spanish. French and Portuguese irony judgments are less crisp and can be driven by specific varieties regardless of annotator origin.

Zero-shot irony detection shows large differences across models and languages. ChatGPT obtains the best results in most languages, while open models often produce weaker and less reliable scores, especially for Arabic and French. The outputs also show that many models do not strictly follow the requested ironic or not ironic labels, which adds another source of uncertainty. ChatGPT is generally stronger than the open models on the positive class, though it performs poorly for Dutch. PolyLM and other open models often stay close to random or fail to identify irony, particularly for Arabic and French. Language prompts and label mapping matter, because models sometimes emit non-standard outputs that must be normalized or counted as incorrect.

The experiments evaluate multilingual irony annotations, sociodemographic disagreement, language-culture alignment, and zero-shot model detection. They show that annotation coverage and label distributions vary across languages, while some demographic dimensions, especially generation, student status, ethnicity, and gender, can produce divergent irony judgments. P-index trends suggest that irony is often easier to recognize when annotators share the relevant language and cultural context, though this effect is clearer for some languages than others. ChatGPT generally outperforms open models, but results remain uneven and model reliability is reduced by inconsistent label outputs.