One-sentence Summary

A study from Bar-Ilan University systematically evaluates n-gram and neural metrics for multilingual abstractive summarization across eight languages from four typological families, showing that n-gram metrics correlate poorly with human judgments for fusional languages, that tokenization considerations can mitigate this morphological sensitivity, and that neural metrics like COMET consistently outperform n-gram metrics in low-resource settings.

Key Contributions

• A large-scale multilingual evaluation suite was built for automatic generation metrics across eight languages from four typological families, using crowdsourced rank annotations and statistical significance reports to analyze metric correlations with human judgments.

• Language typology was shown to affect metric reliability: n-gram metrics correlated less with human assessments for fusional languages than for isolating or agglutinative languages, while tokenization variants could mitigate this problem and even reverse negative correlations for richly morphological fusional languages.

• Evaluation-trained neural metrics, including COMET, correlated more strongly with human judgments than other neural and n-gram metrics, especially in low-resource languages, supporting the use of neural models for multilingual summarization evaluation.

Introduction

The authors address a practical problem for multilingual generative LLMs: as models such as BLOOM, GPT-4o, Gemini, and LLaMA are prompted and deployed in many languages, developers need reliable automatic metrics to judge summarization quality beyond English. Prior evaluation methods struggle because n-gram metrics such as BLEU, ROUGE, and METEOR assume space-delimited, complete-word units, while many languages have rich morphology, ambiguous forms, and non-Latin tokenization needs, and neural metrics such as BERTScore can perform poorly for low-resource or morphologically complex languages. Existing studies also lacked typological breadth, metric diversity, adaptation studies, and statistically reliable correlations with human judgments. The authors leverage about 20,000 human annotations across eight languages from four typological families to systematically compare n-gram and neural metrics, including specialized multilingual evaluators, under different tokenization and lemmatization conditions.

Dataset

• The authors build a human evaluation dataset for abstractive summarization across eight languages, chosen to cover four typological families and include both low-resource and high-resource languages within those groups.

• The underlying source texts and reference summaries are primarily taken from XL-Sum, a multilingual news summarization dataset containing articles paired with human-written summaries. For Hebrew, the authors use HeSum instead.

• For each language, the authors sample 400 articles from the test split. For each sampled article, they generate two parallel summaries, one with GPT-3.5-Turbo 0125 and one with Gemini 1.0 Pro. This gives model-generated summaries paired with the original source article and reference summary information from the source datasets.

• To broaden the quality range of the summaries, the authors artificially corrupt one third of the generated data. Coherence is degraded by replacing nouns and verbs with lemma forms, producing less grammatical sentences, and by reordering non-adjacent sentences to disrupt discourse flow. Completeness is degraded by replacing named entities in summaries with other entities from the source article and by inserting random unrelated sentences.

• The annotation task asks raters to compare a pair of generated summaries while reading the source article. Raters first answer a comprehension question about the article, created with Gemini, to filter out annotations from workers who do not demonstrate understanding.

• Each summary is rated on two quality dimensions, coherence and completeness. Coherence measures grammaticality and readability, while completeness measures how well the summary captures the main ideas of the source text. Ratings use a 1 to 4 Likert scale.

• The dataset is annotated by 36 raters across 13 locales. Raters were recruited through Amazon Mechanical Turk with geographic constraints aligned to the target languages, where possible, and through qualified students who passed a matching questionnaire. For Yoruba and Japanese, native speakers residing outside the expected country were used when local recruitment was not feasible.

• Reliability is controlled by requiring MTurk raters to have more than 100 approved HITs and an approval rate above 90 percent. Each model-generated summary is evaluated by three different raters, and the average score is used for correlation analysis. The authors also report Krippendorff’s alpha, with moderate average agreement of 0.4 for coherence and 0.47 for completeness.

• The resulting dataset can be represented as language-conditioned summary evaluation records containing the source article, two model-generated summaries, the source model identity, whether the summary was clean or corrupted, three human ratings for coherence, three human ratings for completeness, and derived average scores.

• The authors use the dataset to study correlations between automatic evaluation metrics and human judgments, and to produce relative model rankings. Each human pairwise comparison is also converted into Elo-style rankings for model quality by coherence and completeness per language.

Method

The authors design a controlled multilingual evaluation pipeline for assessing automatic summarization metrics against human judgments. The framework is built to capture linguistic and resource-level diversity, covering eight languages from four typological families and pairing high-resource and low-resource languages within each family. Languages are treated as low-resource when their token share in a broad multilingual reference distribution is below $<0.1\%$<0.1%, and high-resource when it reaches $\ge 0.1\%$≥0.1%. This stratification allows the authors to examine whether metric behavior remains consistent across different morphological typologies and levels of data availability.

For each language, the pipeline begins with two parallel candidate summaries generated by large language models from random samples of the corresponding test split. To create a more diverse distribution of quality scores, the authors then synthetically degrade one third of the generated summaries by targeting one evaluation criterion at a time. For coherence, nouns and verbs are replaced with their lemma forms to produce less grammatical text, and nonadjacent sentences are reordered to disrupt discourse flow. For completeness, named entities are substituted with other entities from the source article, and unrelated sentences are inserted. This corruption process intentionally creates summary variants that are not only model-generated but also varied in quality.

The metric evaluation module compares general-purpose scores with multilingual or language-specific variants. The metric pool includes ROUGE with different definitions of wordhood, BERTScore variants that rely on multilingual or monolingual encoders, and COMET as a neural framework for multilingual evaluation. For each language and metric pair, the pipeline computes correlation with human ratings and evaluates statistical significance across the tested dimensions. This systematic pairwise analysis is central to determining whether a metric remains reliable when applied beyond English and across structurally different languages.

Human annotation is incorporated to provide credible reference judgments. Each generated summary is evaluated by three different annotators, and the human score for that item is computed as the mean of the three ratings, $s_{\text{avg}} = \frac{1}{3}\sum_{j=1}^{3}s_j$savg​=31​∑j=13​sj​. Annotators are selected from vetted pools with native or fluent proficiency and are matched to the target languages whenever possible. To reduce noise, annotators complete qualification questions derived from the source article, and only responses from qualified workers are retained. Annotation consistency is measured using Krippendorff’s $\alpha$α for interval ratings, which supports the reliability assessment of the human scores used in the correlation analysis.

Experiment

The experiments build a controlled multilingual evaluation setting for summarization, combining extensive human annotations with generated summaries in eight languages selected to cover different typological families and resource levels. They validate that evaluation reliability depends on language morphology, whether word or token boundaries are handled well, the choice of neural or n-gram metric, and whether the metric is broadly pretrained or specifically trained for generation quality. The findings show that standard n-gram metrics are often unstable for non-English languages unless carefully adapted, while neural metrics, particularly task-specific or monolingually specialized ones, align more closely with human judgments than generic multilingual or large language model judge approaches.

The table categorizes languages across two dimensions: morphological typology and resource availability. It includes isolating, agglutinative, high-fusion, and low-fusion languages, with representative examples in both high-resource and low-resource categories. This grouping supports comparisons across different linguistic structures and data availability conditions. The classification covers four morphological types: isolating, agglutinative, high fusion, and low fusion. Each morphological type is represented by both high-resource and low-resource language examples. High-resource examples include Simplified Chinese, Japanese, Arabic, and Spanish, while low-resource examples include Yoruba, Turkish, Hebrew, and Ukrainian.

The summaries vary widely across language families and resource levels. Low-resource languages generally show higher novel unigram rates than their high-resource counterparts, suggesting more abstract or distilled summaries that may be harder for overlap-based evaluation. High-resource languages tend to achieve stronger compression, while Japanese and Yoruba show higher redundancy and Japanese has notably long mean token lengths. Higher novelty at the unigram level appears mainly in low-resource languages such as Yoruba, Turkish, and Ukrainian. High-resource languages generally compress source articles more than their low-resource counterparts. Redundancy is uneven, with Japanese and Yoruba showing more repeated n-grams than many other languages.

The worker sample is concentrated in the United States, which contains the largest reported group. The remaining participants are spread across a set of smaller country and region groups, with Turkey appearing larger than most of these other entries. Several countries and regions are represented by very small, similar numbers of workers. The United States has the largest worker group in the distribution. Turkey is the largest non-US group reported. Egypt and Jordan appear among the smallest represented groups.

The annotation data indicate moderate human agreement overall, with coherence and completeness judged inconsistently across languages but with a consistent quality gap between the two generated summary sources. Average human scores are broadly comparable, although Arabic is rated lower and Hebrew is rated higher than most other languages. Annotation counts are uneven, with much larger judged samples for Turkish and Chinese than for Japanese, which means some language-level trends are better supported than others. Agreement is strongest for Hebrew and relatively strong for Japanese coherence, while Arabic shows low coherence agreement. The score gap between Gemini and GPT summaries stays close across languages, supporting that the corrupted outputs are separated in quality. Turkish and Chinese have substantially more annotations than Japanese, Hebrew, and Yoruba, creating unequal sample support across languages.

Correlations with human judgments vary by typological family, with n-gram metrics most affected by language morphology. Fusional languages, especially highly fusional ones, show weaker or negative n-gram correlations, while neural metrics are less sensitive to typology and generally correlate better. Adaptations such as multilingual tokenizers, language-specific tokenizers, and lemmatization improve some morphologically rich cases, but can reduce performance for isolating or agglutinative languages in selected criteria. Neural metrics generally align better with human judgments than n-gram metrics. Highly fusional languages are the most difficult for standard n-gram metrics. Morphology-aware or multilingual adaptations can improve correlations for fusional languages.

The experiments evaluate multilingual summarization across languages selected to cover different morphological typologies and resource levels, examining summary properties, annotator distribution, human quality judgments, and metric alignment with human assessments. They find that low-resource languages tend to yield more novel summaries, while high-resource languages generally show stronger compression, and that human ratings consistently distinguish the two generated summary sources despite variable agreement and sample sizes. Annotator coverage is concentrated among U.S. participants, and evaluation correlations reveal that n-gram metrics are especially sensitive to morphology, performing weakest for fusional languages, while neural metrics align more consistently with human judgments.