Research map / Vision-language and multimodal / Human and embodied modeling
Validated Image Caption Rating Dataset
Lothar Narins, Andrew T Scott, Aakash Gautam, Anagha Kulkarni, Mar Castanon, Benjamin Kao, Shasta Ihorn, Yue-Ting Siu, James M Mason, Alexander Mario Blum, Ilmi Yoon
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| rate with VICR 5-level scale and game instead of Flickr8k-Expert scale | dataset · inter-rater agreement | Kendall's W: 84.0 -> 93.8 | Improved |
| rate with VICR 5-level scale and game instead of Flickr8k-Expert scale | dataset · inter-rater agreement | Fleiss kappa: 48.8 -> 58.1 | Improved |
| rate with VICR 5-level scale and game instead of Flickr8k-Expert scale | dataset · inter-rater agreement | Kendall tau X-YZ: 47.7 -> 75.6 | Improved |
| evaluate CIDEr on VICR rather than Flickr8k-Expert | CIDEr · test set | Kendall tau: 44.1 -> 66.6 | Improved |
| evaluate RefCLIPScore on VICR rather than Flickr8k-Expert | RefCLIPScore · test set | Kendall tau: 51.9 -> 71.7 | Improved |
| train rating predictor on VICR instead of Flickr8k-Expert (ablation) | ModelVICR · VICR test | Kendall tau: 71.8 -> 75.8 | Improved |
| train rating predictor on VICR instead of Flickr8k-Expert (ablation) | ModelVICR · Flickr8k-Expert test | Kendall tau: 53.7 -> 53.1 | Hurt |
| use trained ModelVICR instead of CLIPScore | ModelVICR · VICR test | Kendall tau: 67.3 -> 75.8 | Improved |
| raise rater training score by 10% | rater training · simulated rating game | rater competency (logits): +0.17 logits | Improved |
| use reference-free CLIPScore instead of reference-based CIDEr | CLIPScore · Flickr8k-Expert test | Kendall tau: 44.1 -> 50.7 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SPICE — SPICE: Semantic Propositional Image Caption Evaluation (ECCV 2016) · better on 2 of 2 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.84)
- CaptionQA: Is Your Caption as Useful as the Image Itself? (CVPR 2026 · similarity 0.84)
- G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o (AAAI 2025 · similarity 0.83)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.83)
- Q-Eval-100K: Evaluating Visual Quality and Alignment Level for Text-to-Vision Content (CVPR 2025 · similarity 0.83)
- Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
