Research map / Vision-language and multimodal / Human and embodied modeling
NNEval: Neural Network based Evaluation Metric for Image Captioning
Naeha Sharif, Lyndon White, Mohammed Bennamoun, Syed Afaq Ali Shah
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed linear combination of SPICE and CIDEr with a learned neural metric | SPIDEr · COMPOSITE (caption-level correlation) | Pearson correlation: 0.467 -> 0.532 | Improved |
| combine several metric scores with a learned network instead of using SPICE alone | SPICE · COMPOSITE (caption-level correlation) | Pearson correlation: 0.475 -> 0.532 | Improved |
| learned metric instead of SPIDEr on rank correlation | SPIDEr · COMPOSITE (caption-level correlation) | Kendall tau: 0.381 -> 0.404 | Improved |
| learned metric instead of SPIDEr on Spearman | SPIDEr · COMPOSITE (caption-level correlation) | Spearman correlation: 0.495 -> 0.524 | Improved |
| learned composite metric instead of METEOR/SPIDEr | METEOR · PASCAL-50S | average accuracy (%): 78.8 -> 80.5 | Improved |
| learned composite metric on Human-Human Incorrect pairs vs CIDEr | SPIDEr · PASCAL-50S (HI pairs) | accuracy (%): 98.5 -> 99.0 | Improved |
| learned metric vs METEOR on hardest Human-Human Correct pairs | METEOR · PASCAL-50S (HC pairs) | accuracy (%): 61.1 -> 60.4 | Hurt |
| learned metric vs SPICE on Machine-Machine pairs | SPICE · PASCAL-50S (MM pairs) | accuracy (%): 71.6 -> 70.4 | Hurt |
| learned metric instead of SPIDEr under caption distractions | SPIDEr · Distraction tasks (Hodosh-style forced choice) | average accuracy (%): 89.3 -> 91.9 | Improved |
| learned metric raises worst-case accuracy vs SPIDEr | SPIDEr · Distraction tasks (Hodosh-style forced choice) | worst-case accuracy (%): 76.6 -> 85.8 | Improved |
| learned metric vs CIDEr on Replace Scene distractors | CIDEr · Distraction tasks (Replace Scene) | accuracy (%): 95.0 -> 91.8 | Hurt |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SPICE — SPICE: Semantic Propositional Image Caption Evaluation (ECCV 2016) · better on 4 of 5 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- LCEval: Learned Composite Metric for Caption Evaluation (IJCV 2019 · similarity 0.94)
- Learning to Evaluate Image Captioning (CVPR 2018 · similarity 0.92)
- G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o (AAAI 2025 · similarity 0.88)
- CaptionQA: Is Your Caption as Useful as the Image Itself? (CVPR 2026 · similarity 0.88)
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.87)
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
