One-sentence Summary

Researchers from ETH, JHU, MBZUAI, KIT, UvA, QUB, and Charles University introduce the Last Translation Benchmark (LTBv1), a live, peer-reviewed collection of human-authored examples spanning texts, images, audio, and videos that break leading machine translation models, paired with handcrafted verification rules for reliable, actionable evaluation to address benchmark saturation and metric unreliability.

Key Contributions

• Introduces the Last Translation Benchmark (LTBv1), a live, community-contributed dataset of human-authored and peer-reviewed hard examples across texts, images, audio, and videos that break leading machine translation models.
• Presents a new evaluation approach pairing each example with handcrafted verification rules that specify concrete success criteria; passing all rules determines success, enabling an interpretable pass rate and fine-grained failure analysis that distinguishes between poor translations without relying on opaque LLM judge scores.
• Demonstrates utility through examples like English-to-German translation under strict character constraints, where verification rules expose specific failures such as information loss that generic metrics or LLM evaluators would miss, with the dataset structured for ongoing contributions and future releases planned.

Introduction

Machine translation has advanced rapidly and is sometimes claimed to have achieved "lay human parity," yet public trust remains low because a single critical failure can break user confidence. This gap suggests that current research benchmarks are too easy and that evaluation methods fail to reliably detect important errors. Static benchmarks are near saturation and cannot distinguish between strong models, while synthetic benchmarks often produce unnatural inputs that do not reflect real translation use cases. Existing metrics are also flawed: overlap-based metrics misalign with human judgment, trained metrics and LLM judges are biased and vulnerable to hacking, and human evaluation is inconsistent, subjective, and expensive. The authors introduce the Last Translation Benchmark (LTB), a large-scale collection of difficult-to-translate examples across text, images, audio, and video, paired with a new evaluation approach. Each example includes crowdsourced content that most state-of-the-art models struggle with, along with verification rules that target specific failure modes. An LLM judge acts as a verifier, and a translation must pass all rules to succeed, offering a reproducible, interpretable, and cost-effective alternative to generic LLM evaluation. The benchmark currently comprises 3,456 examples across 109 languages and is designed as a live dataset for long-term model benchmarking and diagnosis.

Dataset

The Last Translation Benchmark is a crowdsourced dataset designed to collect and evaluate difficult translation examples. The authors built it through a custom online platform where contributors submit inputs that are hard to translate, either as plain text or with multimodal content such as images, audio, or video.

Dataset composition and sources

• Submissions are collected from registered contributors, who select source and target languages, which can include user-defined forms, dialects, regional variants, or scripts (for example, "Swiss German (Zurich)" or "Serbian (Cyrillic)").
• Each submission includes the input, a human reference translation, and a set of verification rules crafted by the contributor.
• The platform shows up to 10 automatic translations from various models, and contributors inspect these to identify common failure modes and design verification rules.
• An LLM checks each candidate translation against every verification rule. For a submission to be accepted, all but two of ten automatic translations must fail at least one rule, while the human translation must pass.
• Each submission is reviewed by a single reviewer fluent in the relevant languages, who checks that the example is fair, that an expert human translator could pass it, and that the detected errors are perceptible and significant.

Key details for each subset

• LTBv1 contains 3,456 accepted examples collected from May 2026 to September 1st 2026, spanning 109 main languages.
• Language pair distribution: 73% non-English to English, 14% English to non-English, and 13% non-English to non-English.
• Most submissions are textual (94%), with an average length of 19 words or 104 characters.
• Examples come with an average of 1.9 verification rules, and 10% contain translation instructions.
• Language resourcedness (excluding English): 30.5% ultra-high, 28.3% high, 19.0% medium, 3.2% low, 15.2% minimal, and 3.8% zero.
• Language family distribution (excluding English): 58.9% Indo-European, 8.5% Afro-Asiatic, 8.4% Dravidian, 7.5% Sino-Tibetan, and 16.8% other.
• The authors also release LTBv1-eval, a subset of 911 text-only examples selected for highest difficulty, output diversity, and balance across language pairs.

How the paper uses the data

• The dataset is used as a rolling benchmark with tagged releases, and further versions will be released as more data is collected.
• The set of models shown on the interactive platform is kept updated to ensure the benchmark remains challenging and contains non-contaminated data.
• For evaluation, the authors recommend using open-source LLMs as verifiers for reproducibility, though they note that results are stable across different verifiers.
• Performance can be reported on relevant subsets, such as only textual inputs or specific difficulty types, which are annotated in the released dataset.
• A public leaderboard supports two modes: "blind", where the model sees only the input, and "oracle", where the model also sees verification rules, the correct human translation, or other privileged information. The "blind" mode is recommended for comparing realistic translation models.

Processing and metadata construction

• Multimodal examples are handled in two ways: if the media is provided without accompanying text, it is treated as the primary input and translated directly; if text accompanies the media, the media serves as disambiguating context.
• Contributors can specify translation instructions, such as "Use casual language".
• The authors developed a taxonomy of translation difficulty inductively from the collected examples. Two linguists independently annotated a subset, and an LLM was used to scale up annotation to all examples in LTBv1.
• The taxonomy includes categories such as sense-related challenges (polysemy, collocations, style), non-monolingual challenges (false friends, code-mixing, target gaps), non-compositional or creative examples (wordplay, metaphor, meta-reasoning, onomatopoeia), atypical constructions (garden-path sentences, unusual parts of speech), and extralinguistic skills (cultural artifacts, internet cultural artifacts, slang, constraints, and multimodality).
• Each example can be annotated with multiple labels, and these labels are included in the released dataset for filtering and analysis.

Method

Creating the Last Translation Benchmark

The authors build the Last Translation Benchmark through a structured crowdsourcing pipeline that combines human expertise with automated verification to ensure every example is demonstrably difficult for current translation systems. Contributors register on a custom online platform and submit inputs that they consider challenging to translate. Submissions may be textual or multimodal, including images, audio, and video. Accepted submissions are aggregated into a continuously growing rolling benchmark with tagged releases, and contributors with at least ten accepted submissions are invited as dataset co-authors.

The submission process follows a fixed pipeline. First, a contributor selects a source and target language, which may include user-defined forms, dialects, regional variants, or scripts, such as "Swiss German (Zurich)" or "Serbian (Cyrillic)". The contributor then provides the example input, which can be text, audio, image, or video, and writes a correct reference translation. The platform next translates the input using several translation models, and the contributor inspects these automatic translations to identify common failure modes. These observed failures serve as the basis for crafting verification rules. An LLM then checks every candidate translation against each verification rule. To ensure that submissions are demonstrably difficult, a large majority of automatic translations, specifically all but two out of ten, must fail at least one verification rule. At the same time, the provided human translation must pass all rules, demonstrating that it is possible to translate the example well.

Each submission is reviewed by a single reviewer recruited from the pool of contributors who is fluent in the relevant languages. Reviewers check whether the submission follows the guidelines and either approve it or return it for revision with comments. Importantly, reviewers verify that the example is fair, meaning that an expert human translator given the same input would be able to provide a passing translation. They also ensure that the translation errors detected by the verification rules are perceptible and significant failures rather than subtle or debatable issues.

Multimodal inputs receive special handling. If multimodal content is provided without an accompanying text input, such as a picture of a sign with text on it, it is treated as the primary input and translated directly. If multimodal content is accompanied by text, such as a picture of food attached to a social media post, it serves as disambiguating context for the textual input. Contributors can also specify translation instructions, such as "Use casual language," which are incorporated into the verification process.

The platform shows up to ten models interactively, allowing contributors to see model translations of their inputs and observe errors and verification judgments. These models are chosen based on popularity, diversity, and state-of-the-art performance. The specific list of models for a particular example may vary depending on the language pair and whether multimodal support is required. For translation with LLMs, a dedicated prompt is used, and for interactive verification shown to contributors, the authors use Gemini 3.1 Pro with a separate verification prompt. The verification is run for each rule-model pair individually. On average, a contributor makes ten translation attempts before submitting a valid example. The cost of translating a thousand examples is on average only $0.2, while verification is more expensive due to the longer prompt and the need to run against each verification rule, with an average of 1.9 rules per example, costing$0.2,whileverificationismoreexpensiveduetothelongerpromptandtheneedtorunagainsteachverificationrule,withanaverageof1.9rulesperexample,costing1.0 per thousand. The average cost of one accepted example is therefore $0.12.

A Taxonomy of Translation Difficulty

Based on the collected examples, the authors develop a taxonomy of sources of translation difficulty. This taxonomy differs from previous work in translation evaluation by focusing on the source of difficulty in model inputs, whereas prior work generally seeks to categorize translation errors in model outputs. The taxonomy is descriptive and can be interpreted as the set of translation skills needed to process inputs corresponding to various sources of difficulty inherent in examples.

The taxonomy is built inductively. Two linguists independently annotated a subset of examples to understand the types of difficult-to-translate phenomena, using the input, verification rules, and outputs of models on the interactive platform. One example may be annotated with multiple labels. Subsequently, an LLM was used to scale up and annotate all examples in the benchmark.

The largest group of difficulties is labeled "Sense-related," which requires understanding meaning and context and preserving it in translation. This includes polysemy, where a word in the source language has multiple senses that do not carry over to the target language in a given domain. For instance, the domain-specific English word "paper" is not translated correctly to German because the polysemous nature of the source word does not map cleanly. Preserving style and tone remains tricky for models, as shown by examples where sarcasm must be clearly conveyed without being rendered literally. The most mistranslated senses arise from a lack of understanding of collocations and the intensity or gradation of meaning. For example, a Hausa phrase describing torrential rain is mistranslated by models that fail to convey the heavy, continuous nature of the rain.

The "Non-monolingual" group covers difficulties arising from interactions between the source and target languages or from code-mixing. False friends are a clear demonstration, where similar-looking words in different languages are confused by models. Models also lack "target gap" skills when the translation should not translate parts of the source, convert morphemes to words, or expand words to phrases. For example, a Turkish suffix expressing uncertainty requires the English translation to add a word for that uncertainty, and a Japanese borrowing for spam accounts needs to be expanded into a descriptive phrase in Spanish. The source itself can require skills such as teasing apart multiple languages in code-mixed text or capturing nuanced differences in language variants.

"Non-compositional" or creative examples require skills beyond translating literal meaning. Maintaining nuance in poetic language is challenging for even the best models. Wordplay, such as phonological puns, requires understanding the play on spelling and sound, which can be worsened by tokenization issues in low-resource languages. Metaphors require the skill of analogy-making. Meta-reasoning is needed for examples that require thinking about the example itself or working through something about the meaning with reasoning, such as self-referential sentences or preserving ambiguity. A rarer case is onomatopoeia, which requires mapping the word to the sound of its meaning; models often fail to find the target language's equivalent onomatopoeic word.

The last kind of linguistic difficulty is labeled "Atypical," covering non-common, misleading, or adversarial choices of words, phrases, or structures. Examples include sentences without vowels that must be translated while preserving both meaning and the no-vowel constraint, garden-path sentences, and usage of words in atypical parts of speech. These are unusual but meaningful translation challenges, kept separate from other linguistic skills.

Beyond linguistic skills, a second major group requires extralinguistic knowledge that is not language-based, often hard to find in textbooks, and relies on lived language experience. The majority are cultural artifacts known to native speakers, such as using "grandma" in Korean to show respect to elderly strangers, or idioms. Internet cultural artifacts are used by online subcultures, such as using "bread" for crush or borrowing "impression" plus "zombie" for spambots. Other knowledge skills include slang, social conventions like measurement systems, and common named entities.

Extralinguistic skills also include handling constraints that impose restrictions on the translation of an input. These are common in the translation industry, including length constraints in subtitling and computer interfaces. Examples include maintaining the correct output language, specified output length when a shorter paraphrase is needed, preservation of input structure such as dialogue or food menu layouts, and complications arising from the multimodality of input, which may compound textual difficulties and require strong multimodal capabilities.

Experiment

The evaluation introduces the Last Translation Benchmark, a crowdsourced, rolling benchmark of 3,456 difficult translation examples spanning 109 languages, where submissions must fail most automatic translations while passing human-written verification rules. Results show that state-of-the-art models achieve low verifier pass rates, but performance surges when models are given the verification rules, indicating they can satisfy challenges they fail to identify independently. The verification-rule-based evaluation is more stable across LLM verifiers, better aligned with human judgments, and less prone to self-bias than generic LLM judges or standard metrics. A grounded taxonomy of translation difficulty reveals classic challenges like polysemy and metaphor, alongside underexplored ones such as meta-reasoning, internet cultural artifacts, and phonological wordplay, with models also exhibiting blockers like hallucinations, refusals, and instruction injection.

The dataset comprises 3,456 accepted examples across 109 main languages, with English-centric pairs dominating, though a notable 13% of examples are non-English to non-English. The benchmark is designed as a stress test, featuring diverse linguistic challenges such as idioms, syntactic ambiguities, and pragmatic nuances, with verification rules and linguistic annotations to track model failures. English-to-non-English pairs account for 14% of examples, while non-English-to-English pairs are the majority at 73%. Non-English-to-non-English pairs make up 13% of the dataset, indicating a deliberate inclusion of language pairs beyond English. The benchmark includes examples with translation instructions (10%) and an average of 1.9 verification rules per example. Linguistic challenges span phonological, morphological, syntactic, lexical/semantic, pragmatic, and orthographic levels. Model failures include irrelevant outputs, refusals due to perceived toxicity, incomplete translations, and instruction injection issues.

The table compares model performance on the Last Translation Benchmark using multiple evaluation methods, including rule-based verification, LLM-as-a-judge, automated metrics, and human judgment. Open-weight models and models shown to contributors are marked, with results averaged across the eval set. The verifier pass rate is highlighted as the official performance measure, while other metrics tend to overestimate translation quality. The verifier pass rate is the official metric and is more stable across evaluator model choices than generic LLM judges or neural metrics. Generic LLM-as-a-judge scores correspond to 'good' translations despite low verifier pass rates, indicating a disconnect between perceived quality and rule compliance. Automated metrics fail to capture improvements when models are given verification rules or when human translations avoid listed failure cases. The verifier pass rate agrees better with human judgments than other evaluation methods.

Providing LLMs with human-written verification rules in their prompts leads to a substantial increase in translation performance, as measured by the verifier pass rate, while synthetic rules generated by the models themselves yield only a slight improvement. This suggests that LLMs can often satisfy explicit constraints but struggle to identify the specific challenges in a translation problem on their own. Human-written verification rules dramatically boost verifier pass rates compared to no rules. Synthetic rules generated by the LLMs produce only a marginal improvement, far below the performance with human rules. The improvement with human rules indicates that LLMs can follow explicit constraints but fail to infer the required translation challenges independently.

The table compares ranking similarity across three evaluation approaches: verifier-based, LLM judge, and neural metrics. Verifier-based evaluation shows high stability across model choices and strong agreement with human rankings, while generic judges and neural metrics are less stable and agree less with human judgments. Verifier-based rankings are highly stable across different evaluator models, with similarity around 86.9. Generic judge and neural metric rankings are less stable, with self-similarities around 71.3 and 35.1 respectively. Verifier-based rankings agree much more with human rankings (90.5) than judge or metric rankings do (34.9 and 16.2). Cross-approach agreement is low, indicating that different evaluation methods produce divergent model rankings.

The table measures model self-bias as the difference between a model's self-ranking and its ranking by other LLMs, with positive values indicating self-preference. Across the listed models, verifier-based self-bias is consistently lower than judge-based self-bias, suggesting that using verification rules reduces evaluator subjectivity. The largest self-preference appears in judge settings, while some models show slight self-dispreference in both conditions. Verifier-based self-bias values are markedly lower than judge-based values for every model listed. Gemma 4 shows the highest self-preference in the judge setting, while Gemini 3.1 Pro is the only model with negative self-bias in both conditions. The gap between verifier and judge self-bias is largest for Gemma 4 and Qwen 3.7 Flash, indicating that verification rules help mitigate evaluator self-preference. Most models exhibit positive self-preference, but the magnitude is reduced when using verification rules, supporting the objectivity of the verifier approach.

The benchmark evaluates 3,456 translation examples across 109 languages, with English-centric pairs dominating and 13% non-English to non-English pairs, and includes linguistic challenges spanning phonological to orthographic levels. The verifier pass rate, based on human-written verification rules, is the official metric and shows stronger stability across evaluator models and better agreement with human judgments than generic LLM judges or neural metrics, which tend to overestimate quality. Providing LLMs with human-written verification rules substantially boosts verifier pass rates, while synthetic rules yield only marginal gains, indicating models can follow explicit constraints but struggle to infer challenges independently. Verifier-based rankings are also more stable and agree more with human rankings than judge or metric rankings, and verifier-based self-bias is consistently lower than judge-based self-bias, suggesting verification rules reduce evaluator subjectivity.