Research map / Vision-language and multimodal / Multimodal LLMs
G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes G-VEval, a GPT-4o-based chain-of-thought captioning metric with expected-score computation and three reference/visual modes, plus the MSVD-Eval video caption dataset with ACCR criteria, reporting higher human correlation than prior metrics.
Problem
Automatic evaluation of image and video captions is weak: n-gram metrics miss semantics, trained metrics fail zero-shot, and existing LLM-based metrics like CLAIR are reference-only and lack interpretability, while video caption evaluation lacks clear criteria.
Approach
Adapt G-Eval to multimodal captioning by prompting GPT-4o with evaluation criteria, CoT steps, visual input (images or three concatenated video frames), and references, requiring a written reason before the score, and computing the expected score from logprobs; it supports reference-free, reference-only, and combined modes.
Key results
- On Flickr8k-Expert, G-VEval-ref-free reaches Kendall tau-b 61.5 and tau-c 59.7, above CLAIR (58.3 / 48.8) and Polos (56.4).
- On MSVD-Eval (averaged ACCR scores), G-VEval-combined achieves tau-b 62.9 and tau-c 63.7, versus 44.6 / 40.3 for CLAIR.
- On VATEX-EVAL with 9 references, G-VEval reaches 48.1 versus 34.8 for CLAIR and 36.8 for EMScore.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use visual content only rather than references only or references plus visual content | G-VEval · Flickr8k-Expert and Flickr8k-CF | Kendall τ_b: Flickr8k-Expert: 60.4 -> 61.5; Flickr8k-CF: 37.2 -> 38.7 | Mixed |
| Use references plus visual content rather than visual content only | G-VEval · MSVD-Eval | Kendall τ_b: 59.6 -> 62.9 | Helped |
| Use references plus visual content rather than visual content only | G-VEval · Flickr8k-Expert and Flickr8k-CF | Kendall τ_b: Flickr8k-Expert: 61.5 -> 60.5; Flickr8k-CF: 38.7 -> 38.2 | Hurt |
| Use references plus visual content rather than visual content only | G-VEval · VATEX-EVAL | Human judgment correlation: 39.4 -> 44.9 -> 48.1 | Helped |
| Use 0-to-100 scoring instead of 1-to-5 rating | G-VEval · Flickr8k-Expert | Kendall τ_b: 54.393 -> 60.385 | Helped |
| Use expected score calculation instead of GPT-4o's single score (ablation) | G-VEval · Flickr8k-Expert | Kendall τ_b: 59.118 -> 60.385 | Helped |
| Use expected score calculation instead of GPT-4o's single score (ablation) | G-VEval · Flickr8k-Expert | Kendall τ_c: 54.847 -> 58.598 | Helped |
| Add CoT evaluation steps (ablation) | G-VEval · Flickr8k-Expert | Kendall τ_b: 50.157 -> 60.385 | Helped |
| Add CoT evaluation steps (ablation) | G-VEval · Flickr8k-Expert | Kendall τ_c: 48.408 -> 58.598 | Helped |
| Include a detailed reason in the response (ablation) | G-VEval · Flickr8k-Expert | Kendall τ_b: 52.436 -> 60.385 | Helped |
| Include a detailed reason in the response (ablation) | G-VEval · Flickr8k-Expert | Kendall τ_c: 26.944 -> 58.598 | Helped |
Surprising findings
- Removing the explicit reasoning from the response drops Kendall tau-c on Flickr8k-Expert from 58.6 to 26.9, despite a smaller tau-b drop.
- The reference-free setting outperforms the combined setting on Flickr8k-Expert, suggesting noisy reference captions can hurt evaluation.
Limitations
- Relies on GPT-4o, which is costly and closed; future model updates may change scoring consistency.
- Designed for short videos (under 10 seconds); longer videos may need scene detection.
- MSVD-Eval is small (150 clips per the main text, 200 in the appendix) and uses candidate captions from a single model.
Experimental setup
Datasets: Flickr8k-Expert Flickr8k-CF VATEX-EVAL MSVD-Eval MSVD validation set
Benchmarks: Flickr8k-Expert Flickr8k-CF VATEX-EVAL MSVD-Eval
Metrics: Kendall tau-b Kendall tau-c
Baselines: BLEU ROUGE METEOR CIDEr SPICE BERTScore CLIP-S PAC-S RefPAC-S Polos CLAIR EMScore
Builds on: G-Eval CLAIR EMScore PAC-S
Closest papers by meaning
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.90)
- Distilling Vision-Language Models on Millions of Videos (CVPR 2024 · similarity 0.88)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.88)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.87)
- Vript: A Video Is Worth Thousands of Words (NeurIPS 2024 · similarity 0.87)
- CaptionQA: Is Your Caption as Useful as the Image Itself? (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.