One-sentence Summary

Columbia University and EPFL researchers show that large language models can generate synthetic humor detection data by unfunning jokes, with LLM-produced non-humorous counterparts serving as challenging adversarial examples for classifiers, and extend this approach to code-mixed English-Hindi humor datasets.

Key Contributions

• The paper benchmarks large language models on the human-curated Unfun task and finds that models such as GPT-4 and GPT-3.5 can outperform human editors at removing humor from texts.

• The paper repurposes model-generated unfunny edits as synthetic paired data for humor detection, demonstrating that these edits can support training humor classifiers and expand aligned humorous and non-humorous examples.

• The paper extends the approach to a code-mixed English-Hindi humor dataset, where GPT-4 generated unfunny variants are rated highly by bilingual annotators and provide challenging adversarial examples for classifiers trained on the original corpus.

Introduction

Large language models perform well on many language tasks but still struggle to detect, explain, and generate humor reliably, a limitation that matters as AI systems are used more often in social, creative, and safety-sensitive settings. Prior computational humor work relies heavily on human-curated datasets such as the Unfun corpus, and although useful, those resources are limited in size and scope because high-quality annotation is difficult and expensive. The authors leverage the asymmetry between creating humor and removing humor, showing that models such as GPT-4 and GPT-3.5 can outperform humans at editing humorous headlines into unfunny versions. They use this capability to generate large synthetic paired datasets for training humor classifiers and extend the method to a code-mixed English-Hindi dataset, where the synthetic examples are rated highly by bilingual annotators and provide challenging test data.

Dataset

• The paper builds on the Unfun dataset, originally created from human gameplay where players edited satirical The Onion headlines into serious-looking headlines.

• The main Unfun subset contains approximately 11K satirical-to-serious headline pairs, with a smaller subset rated by players.

• The authors use these satirical/unfunned headline pairs as a benchmark to compare LLMs with human players at removing humor while preserving the original text structure.

• The dataset construction relies on a two-sided reward signal: the edited headline should remain token-level similar to the original satire and be judged convincingly serious by other players.

• The authors also extend the dataset to code-mixed English-Hindi Twitter data, starting from tweets that were previously annotated as humorous or non-humorous.

• For this extended subset, GPT-4 is prompted to unfun humorous tweets, and the generated outputs are filtered by removing examples that GPT-4 still classifies as humorous.

• Bilingual human annotators then rate the GPT-4-generated unfunned tweets alongside original samples to assess quality and humor removal.

• The authors further use a held-out set of synthetic tweets rated as coherent and successfully non-humorous to test humor classifiers trained with varying proportions of synthetic non-humorous data.

Method

The unfun generation pipeline is designed to transform humorous or satirical text into more literal, serious text. The overall approach combines two complementary strategies: few-shot large language model prompting and a compact probabilistic token-editing module.

In the few-shot generation setting, each input humorous text is provided to a language model together with a short task description and a small number of exemplar pairs. Each exemplar consists of an input humorous text and a corresponding output serious text. The exemplars are not selected arbitrarily; they are sampled from a subset of existing human-rated examples that have been judged high-quality. This sampling strategy is intended to encourage diversity in the generated serious rewrites while keeping the style and task format consistent across examples. The pipeline is evaluated with several instruction-following and non-instruct language models, including large closed-source models and open-source models of smaller size. The central assumption is that, given enough prompt structure and representative demonstrations, the language model can act as a stylistic rewriter that suppresses satirical or humorous signals.

The second component is a lightweight token-editing method based on a masked language model. This approach is motivated by the observation that humor is often associated with unexpectedness, and that humorous headlines can exhibit higher perplexity. The method, denoted ROBERTA-SWAP, uses a pretrained masked language model to identify and replace tokens that appear unlikely in the local context of the headline. Given a headline sequence $x = (x_1, x_2, \dots, x_n)$x=(x1​,x2​,…,xn​), the model considers each position $i$i by masking the original token $x_i$xi​ and computing the probability distribution over possible replacements. Let

be the probability assigned to the original token at position $i$i, and let

be the most likely replacement token predicted by the masked language model. The replacement probability is then

Positions are ranked using a ratio between the original token probability and the predicted replacement probability, for example

Positions with a strong mismatch between the original token and the model’s most likely replacement are considered candidates for editing. From these candidates, the pipeline selects $k$k positions and iteratively replaces each selected token with its highest-probability masked-language-model prediction.

Conceptually, ROBERTA-SWAP performs a targeted smoothing of the headline at the token level. Rather than rewriting the entire text through generation, it focuses on local elements that introduce surprise or unusual probability mass. By replacing such tokens with more predictable alternatives, the method produces a more literal version of the original text. This design is computationally lighter than full generative editing, since it relies on masked scoring rather than autoregressive decoding, and it directly operationalizes the connection between humor, surprise, and token-level likelihood.

Experiment

The experiments evaluate synthetic Unfun data using automatic humor classification and human ratings, comparing LLM-generated humor and unfunny edits with human satire and crowd worker edits. Results indicate that LLMs are weak at generating human-quality humor but can reliably edit away humor, sometimes outperforming crowd workers and yielding aligned unfun data that helps detectors. Additional English and Hindi code mixed evaluations show this editing capability generalizes, though classifiers remain sensitive to superficial cues and subjective humor judgments complicate evaluation.

Across the two editing directions, removing humor is easier than generating it with language models. Synthetic unfunny data, especially from GPT-4, helps humor classifiers reach the strongest holdout performance and remains close to human-edited data, while real news headlines perform poorly. Synthetic humor remains weaker than human-edited data or unfunny baselines, even though edit distances indicate that the original humorous content is largely retained. GPT-4 unfunny examples support the best classifier accuracy in both directions Aligned synthetic data beats real news headlines but synthetic humor lags behind unfunny baselines

Human evaluations show that model-generated satirical humor remains less funny and less convincing than original Onion headlines. At the same time, generated unfunny edits are often judged as real news, while remaining grammatical, coherent, and similar in unfunniness to human-edited examples. GPT-3.5 and GPT-4 generated humor without a clear quality gap between them. Synthetic unfun examples were more likely than human-edited unfuns to be rated as real news. ROBERTA-SWAP matched human players on funniness and real-headline ratings but lagged in coherence.

Human evaluations show that GPT-4 edited English-Hindi tweets are rated as less humorous than originally humorous tweets and similarly to nonhumorous tweets, while maintaining high coherence. Applying a GPT-4 filter further lowers humor ratings but produces fewer accepted samples and somewhat lower coherence. The results support that GPT-4 can convert humorous code-mixed tweets into nonhumorous text, though it requires more edits than in the English satirical headline setting. GPT-4 edits substantially reduce humor ratings for originally humorous tweets while keeping coherence high. GPT-4 filtering creates a smaller set judged as much less humorous, at a modest coherence cost. Nonhumorous human tweets and GPT-4 unfun tweets receive similar low humor ratings.

Humor classifiers trained on the original English-Hindi tweets generalize poorly to human-vetted unfunned samples, indicating that they rely on surface cues rather than robust humor understanding. Adding synthetic unfunned examples to training substantially raises accuracy on those adversarial samples, but this improvement comes with reduced accuracy on humorous and balanced original examples. Models trained only on original data show poor performance on unfunned tweets. Including synthetic unfunned data yields the highest accuracy on unfunned samples. Training with more synthetic unfuns lowers original-dataset balanced accuracy and humor accuracy.

The experiments evaluate humor generation and removal using synthetic edits of satirical English headlines and English-Hindi tweets, alongside classifier training and human judgments. Removing humor is easier and more reliable than generating it: GPT-4 synthetic unfunny examples are close to human edits and strengthen classifier performance, while synthetic humor is less funny and less convincing than human originals. Human evaluations further show that generated unfunny edits can resemble real news and retain coherence, but classifiers trained only on original data rely on superficial cues, with synthetic unfunniness improving adversarial robustness at the cost of some original-data accuracy.