Research map / Vision-language and multimodal / Human and embodied modeling
BERTScore: Evaluating Text Generation with BERT
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace exact n-gram overlap with contextual token-embedding similarity and greedy token matching | BERTScore · WMT18 segment-level machine translation | Kendall correlation: .415/.620 -> .550/.728 | Helped |
| Add IDF importance weighting (ablation) | BERTScore F1 · WMT18 machine translation | human correlation | Mixed |
| Add IDF importance weighting (ablation) | BERTScore recall · COCO 2015 Captioning Challenge | Pearson correlation M1: 0.888 -> 0.917 | Helped |
| Add IDF importance weighting (ablation) | BERTScore F1 · PAWS-QQP | AUC: 0.685 -> 0.693 | Helped |
| Use an intermediate rather than final contextual-encoder layer (ablation) | BERTScore · WMT16 segment-level machine translation | human correlation | Helped |
| Add power means to aggregate representations from different layers (ablation) | BERTScore · WMT17 to-English segment-level machine translation | Pearson correlation | Mixed |
| Use a BERT model fine-tuned on MNLI (ablation) | BERTScore · WMT17 to-English segment-level machine translation | Pearson correlation | Helped |
| Recompute IDF scores on candidate sentences (ablation) | BERTScore · WMT17 to-English segment-level machine translation | Pearson correlation | Mixed |
| Exclude punctuation and all but the first subword token from matching (ablation) | BERTScore · WMT17 to-English segment-level machine translation | Pearson correlation | Mixed |
| Replace greedy token matching with optimal transport matching (ablation) | BERTScore · WMT17 to-English segment-level machine translation | Pearson correlation | Mixed |
| Use contextual token similarity rather than lexical overlap | BERTScore · PAWS-QQP adversarial paraphrase classification | AUC: 0.527 -> 0.693 | Helped |
| Use BERTScore recall instead of precision or F1 | BERTScore · MSR Abstractive Text Compression meaning judgments | Pearson correlation: 0.58 -> 0.64 | Helped |
Closest papers by meaning
- BARTScore: Evaluating Generated Text as Text Generation (NeurIPS 2021 · similarity 0.91)
- SMART: Sentences as Basic Units for Text Evaluation (ICLR 2023 · similarity 0.87)
- RevisEval: Improving LLM-as-a-Judge via Response-Adapted References (ICLR 2025 · similarity 0.85)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.85)
- Generative Judge for Evaluating Alignment (ICLR 2024 · similarity 0.84)
- Retrieval is Accurate Generation (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.