Yang Wang Chenghao Xiao Chia-Yi Hsiao Zi Yan Chang Chi-Li Chen Tyler Loakman Chenghua Lin

Abstract

We introduce Drivelology, a unique linguistic phenomenon characterised as"nonsense with depth", utterances that are syntactically coherent yetpragmatically paradoxical, emotionally loaded, or rhetorically subversive.While such expressions may resemble surface-level nonsense, they encodeimplicit meaning requiring contextual inference, moral reasoning, or emotionalinterpretation. We find that current large language models (LLMs), despiteexcelling at many natural language processing (NLP) tasks, consistently fail tograsp the layered semantics of Drivelological text. To investigate this, weconstruct a small but diverse benchmark dataset of over 1,200 meticulouslycurated examples, with select instances in English, Mandarin, Spanish, French,Japanese, and Korean. Annotation was especially challenging: each of theexamples required careful expert review to verify that it truly reflectedDrivelological characteristics. The process involved multiple rounds ofdiscussion and adjudication to address disagreements, highlighting the subtleand subjective nature of the Drivelology. We evaluate a range of LLMs onclassification, generation, and reasoning tasks. Our results reveal clearlimitations of LLMs: models often confuse Drivelology with shallow nonsense,produce incoherent justifications, or miss the implied rhetorical functionaltogether. These findings highlight a deeper representational gap in LLMs'pragmatic understanding and challenge the assumption that statistical fluencyimplies cognitive comprehension. We release our dataset and code to facilitatefurther research in modelling linguistic depth beyond surface-level coherence.