One-sentence Summary

Researchers from Cornell University and ASAPP Inc. propose BERTScore, an automatic text-generation evaluation metric that computes token similarity using contextual BERT embeddings instead of exact matches, and show that it correlates better with human judgments, selects models more effectively across 363 machine translation and image captioning systems, and remains more robust on adversarial paraphrase detection than existing metrics.

Key Contributions

• BERTSCORE is introduced as an automatic text generation evaluation metric that computes token similarity using contextual BERT embeddings instead of exact matches, allowing relaxed matching and optional corpus-based importance weighting.
• Evaluation on outputs from 363 machine translation and image captioning systems shows BERTSCORE correlates better with human judgments and provides stronger model selection performance than existing metrics.
• An adversarial paraphrase detection task shows that BERTSCORE is more robust to challenging examples than existing metrics.

Introduction

Automatic evaluation of natural language generation, such as machine translation and image captioning, requires measuring semantic equivalence between candidate texts and references. Widely used metrics like BLEU and METEOR rely on surface-form n-gram overlap or heuristic matching, so they often fail to robustly match paraphrases and can miss semantically important ordering changes. The authors introduce BERTScore, an evaluation metric that uses pretrained BERT contextual embeddings to compute sentence similarity as the sum of cosine similarities between token embeddings. This approach captures meaning-preserving lexical variation and distant dependencies without task-specific supervision, and the authors show it correlates more strongly with human judgments than existing metrics.

Method

The authors propose BERTSCORE, a metric that evaluates text generation by leveraging contextual embeddings to compute token-level similarity between a reference sentence and a candidate sentence. Instead of relying on exact string matching or static word embeddings, the method utilizes contextual embeddings such as BERT. These models generate distinct vector representations for identical words based on their surrounding context. The input text is tokenized into word pieces, and a Transformer encoder computes the final representation for each token through alternating self-attention and nonlinear transformations.

Once the tokens are embedded, the framework computes a soft similarity measure. The authors use cosine similarity between reference and candidate token vectors. By employing pre-normalized vectors, this calculation simplifies to an inner product. To compute the final scores, the method applies greedy matching, pairing each token in one sentence with its most similar counterpart in the other sentence.

As shown in the figure below, the complete pipeline processes the reference and candidate sentences through contextual embedding, pairwise cosine similarity calculation, and maximum similarity matching.

Recall, precision, and F1 scores are derived from these matched similarities. The recall $R_{\text{BERT}}$RBERT​ and precision $P_{\text{BERT}}$PBERT​ are defined as:

$$R_{\text{BERT}} = \frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} \mathbf{x}_i^\top \hat{\mathbf{x}}_j, \quad P_{\text{BERT}} = \frac{1}{|\hat{x}|} \sum_{\hat{x}_j \in \hat{x}} \max_{x_i \in x} \mathbf{x}_i^\top \hat{\mathbf{x}}_j$$RBERT​=∣x∣1​xi​∈x∑​x^j​∈x^max​xi⊤​x^j​,PBERT​=∣x^∣1​x^j​∈x^∑​xi​∈xmax​xi⊤​x^j​

The final F1 score $F_{\text{BERT}}$FBERT​ combines these two metrics harmonically.

To enhance metric sensitivity, the authors introduce an optional importance weighting mechanism using inverse document frequency scores computed from the test corpus. This ensures that rare, highly indicative words contribute more to the final score than common words. The idf-weighted recall is formulated as:

$$R_{\text{BERT}} = \frac{\sum_{x_i \in x} \operatorname{idf}(x_i) \max_{\hat{x}_j \in \hat{x}} \mathbf{x}_i^\top \hat{\mathbf{x}}_j}{\sum_{x_i \in x} \operatorname{idf}(x_i)}$$RBERT​=∑xi​∈x​idf(xi​)∑xi​∈x​idf(xi​)maxx^j​∈x^​xi⊤​x^j​​

Finally, because pre-normalized vectors constrain the raw cosine similarity scores to a narrow numerical range, the authors implement a baseline rescaling technique to improve score readability. They compute an empirical lower bound $b$b by averaging the BERTSCORE of randomly paired sentences from a large monolingual corpus. The raw scores are then linearly rescaled using this baseline, mapping the values closer to a standard 0 to 1 range without altering the ranking capability or human correlation of the metric.

Experiment

The experiments evaluate BERTScore on machine translation using WMT datasets and on image captioning using the COCO 2015 Captioning Challenge, with additional robustness tests on adversarial paraphrase classification. In machine translation, BERTScore consistently ranks among the top metrics for system-level and segment-level correlation with human judgments and for model selection, with segment-level improvements over BLEU being particularly notable. For image captioning, it outperforms all task-agnostic baselines by large margins, and idf weighting proves beneficial. The robustness analysis shows BERTScore retains strong paraphrase detection performance on challenging PAWS adversarial examples while other metrics degrade substantially.

BERTScore variants, particularly F_BERT, generally achieve strong system-level Pearson correlations with human judgments across most WMT18 language pairs, often matching or surpassing BLEU, ITER, and YiSi-1. RUSE is competitive in to-English settings but absent from from-English results because trained models were not available. The Turkish to-English setting is a notable exception where BERTScore correlations are lower and more variable. F_BERT is consistently among the strongest metrics across most language pairs and directions, generally matching or exceeding BLEU, ITER, and YiSi-1. RUSE is competitive for to-English but lacks from-English results because trained models were not available.

Across WMT18 system-level correlations, BERTScore variants are generally among the strongest metrics, with F_BERT matching or exceeding BLEU and ITER on most language pairs and directions. RUSE is competitive for to-English settings but lacks from-English results. The Turkish pair is a notable exception where correlations drop for several metrics and YiSi-1 performs especially well. F_BERT tends to match or exceed BLEU and ITER correlations across most language pairs and translation directions. RUSE reports strong to-English correlations but no from-English values, while YiSi-1 is particularly strong on the Turkish pair.

Across WMT18 hybrid system model selection, BERTScore variants are consistently among the strongest metrics, with F_BERT leading or near-leading on most to-English directions and several from-English directions. RUSE is competitive for to-English evaluation but is unavailable for from-English settings, where BERTScore and YiSi-1 provide the best alternatives. Overall, no single metric dominates every language pair and direction. F_BERT is the top model selection metric in most to-English directions, and BERTScore variants also lead or tie on several from-English pairs. RUSE achieves the highest to-English accuracy for some language pairs but has no from-English results, leaving BERTScore and YiSi-1 as the leading metrics in those directions.

Segment-level results on WMT18 show BERTScore variants generally achieve the strongest Kendall correlations with human judgments, for both to-English and from-English directions across all language pairs. RUSE is competitive only in to-English, where its supervised models are available, while BLEU and ITER trail the other metrics. The gap between BERTScore and BLEU is especially pronounced, suggesting BERTScore is more reliable for example-level analysis. BERTScore variants are top performers across nearly all language pairs and translation directions. F_BERT and R_BERT are usually the strongest BERTScore configurations at segment level. RUSE performs competitively for to-English but is unavailable for from-English, limiting its applicability. BLEU and ITER show consistently lower segment-level correlations, with BLEU lagging far behind BERTScore.

On the COCO Captioning Challenge, the learned image-aware metric LEIC achieves the highest correlation with human judgments, while n-gram metrics such as BLEU and ROUGE-L correlate poorly. SPICE is the strongest among the listed task-agnostic metrics, and the paper reports that BERTSCORE outperforms all task-agnostic baselines by large margins. LEIC, which uses images as additional inputs and is optimized for COCO, has the highest reported correlations. BLEU and ROUGE-L show weak or near-zero correlations, suggesting strict n-gram matching is unreliable for image captioning evaluation. SPICE has the strongest correlations among the listed non-image metrics. The paper notes that BERTSCORE outperforms all task-agnostic baselines by a large margin on this task.

The experiments evaluate BERTScore on WMT18 system-level correlation, hybrid system model selection, segment-level correlation, and COCO image captioning. BERTScore variants, especially F_BERT, are consistently among the strongest metrics and often match or exceed BLEU, ITER, and YiSi-1 across most language pairs and directions, with Turkish being a notable exception and RUSE limited to to-English settings due to unavailable models. Segment-level results show BERTScore leading clearly while BLEU and ITER lag behind, supporting its reliability for example-level analysis. On COCO, BERTScore outperforms task-agnostic baselines by a large margin, while n-gram metrics correlate poorly and the image-aware LEIC performs best.