Research map / Vision-language and multimodal / Human and embodied modeling
SMART: Sentences as Basic Units for Text Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use sentence-level soft matching with BLEURT rather than token-level baseline metrics | SMART on SummEval · SummEval | average system-level Kendall tau across quality dimensions: .619 vs .511 (BARTScore+CNN) | Helped |
| use sentence-level matching with CHRF rather than token-level CHRF | SMART on SummEval · SummEval | average system-level Kendall tau across quality dimensions: .523 vs .519 | No clear effect |
| add both precision and recall through the f-measure instead of using precision only (ablation) | Sₓ-BLEURT · SummEval | average system-level Kendall tau across quality dimensions: .177 -> .628 | Helped |
| add both precision and recall through the f-measure instead of using recall only (ablation) | Sₓ-BLEURT · SummEval | average system-level Kendall tau across quality dimensions: .440 -> .628 | Helped |
| include source comparison alongside reference comparison (ablation) | Sₓ-BLEURT · SummEval | average system-level Kendall tau across quality dimensions: -.131 -> .628 | Helped |
| include reference comparison alongside source comparison (ablation) | Sₓ-BLEURT · SummEval | average system-level Kendall tau across quality dimensions: .465 -> .628 | Helped |
| take the maximum of source and reference scores instead of averaging them (ablation) | Sₓ-BLEURT · SummEval | average system-level Kendall tau across quality dimensions: .582 -> .628 | Helped |
| take the maximum of source and reference scores instead of their minimum (ablation) | Sₓ-BLEURT · SummEval | average system-level Kendall tau across quality dimensions: .465 -> .628 | Helped |
| replace the CNN/DM reference with the best system summary and retain source/reference maximum aggregation (ablation) | Sₓ-BLEURT · SummEval | average system-level Kendall tau across quality dimensions: .661 -> .686 | Helped |
Closest papers by meaning
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.85)
- RevisEval: Improving LLM-as-a-Judge via Response-Adapted References (ICLR 2025 · similarity 0.85)
- Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation (AAAI 2025 · similarity 0.85)
- Generative Judge for Evaluating Alignment (ICLR 2024 · similarity 0.85)
- Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation (NeurIPS 2025 · similarity 0.84)
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.