Research map / Vision-language and multimodal / Human and embodied modeling
SPICE: Semantic Propositional Image Caption Evaluation
Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SPICE instead of CIDEr for judging captions | SPICE · COCO 2015 Challenge system-level | Pearson correlation with human M1: 0.43 -> 0.88 | Improved |
| use SPICE instead of METEOR for judging captions | SPICE · COCO 2015 Challenge system-level | Pearson correlation with human M1: 0.53 -> 0.88 | Improved |
| add synonym matching to exact-match SPICE (ablation) | SPICE · COCO 2015 Challenge system-level | Pearson correlation with human M1: 0.84 -> 0.88 | Improved |
| use SPICE instead of CIDEr on detail judgments | SPICE · COCO 2015 Challenge system-level | Pearson correlation with human M4 (detail): 0.46 vs -0.21 | Improved |
| use SPICE instead of CIDEr at caption level on Flickr 8K | SPICE · Flickr 8K caption-level | Kendall tau with human: 0.44 -> 0.45 | Improved |
| use SPICE instead of CIDEr at caption level on the composite dataset | SPICE · Composite dataset caption-level | Kendall tau with human: 0.36 -> 0.39 | Improved |
| use SPICE instead of METEOR on pairs of model captions | SPICE · PASCAL-50S (5 references), MM pairs | accuracy (%): 66.8 -> 68.2 | Improved |
| use SPICE instead of METEOR on all PASCAL-50S pairs | SPICE · PASCAL-50S (5 references), all pairs | accuracy (%): 80.8 -> 78.8 | Hurt |
| use SPICE instead of METEOR on human-vs-model pairs | SPICE · PASCAL-50S (5 references), HM pairs | accuracy (%): 94.2 -> 87.5 | Hurt |
Later papers that compared with this method
7 later accepted papers report SPICE as a baseline in their results tables; 7 of them report a better number on most of the results they share with it.
- PoSh: Using Scene Graphs to Guide LLMs-as-a-Judge for Detailed Image Descriptions (ICLR 2026 · better on 2 of 2 shared results)
- Image Clustering Conditioned on Text Criteria (ICLR 2024 · better on 3 of 3 shared results)
- Validated Image Caption Rating Dataset (NeurIPS 2023 · better on 2 of 2 shared results)
- BERTScore: Evaluating Text Generation with BERT (ICLR 2020 · better on 1 of 1 shared result)
- Towards Unique and Informative Captioning of Images (ECCV 2020 · better on 1 of 1 shared result)
- NNEval: Neural Network based Evaluation Metric for Image Captioning (ECCV 2018 · better on 4 of 5 shared results)
- Learning to Evaluate Image Captioning (CVPR 2018 · better on 3 of 3 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- CaptionQA: Is Your Caption as Useful as the Image Itself? (CVPR 2026 · similarity 0.89)
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.89)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.87)
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.86)
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.86)
- G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
