Research map / Vision-language and multimodal / Human and embodied modeling
Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CompreCap, a human-annotated directed scene graph benchmark (560 MSCOCO-derived images) for evaluating detailed image captions from LVLMs via object, attribute, and relation-level matching using Llama3 as an LLM judge.
Problem
Existing caption benchmarks and metrics (short-caption datasets, n-gram metrics, CLIPScore, and DetailCaps' isolated-term parsing) fail to evaluate detailed LVLM captions for accuracy and comprehensiveness, because they break the binding between objects, attributes and directional relations.
Approach
Human-annotate images with segmentation masks, object categories, bound attribute descriptions and directed subject-verb-object relations forming a directed scene graph, then score generated captions hierarchically (object coverage, attribute score, relation score, pixel coverage) with an LLM matching sub-captions to annotations, plus a tiny-object VQA task.
Key results
- LLaVA-Next-34B (Sunified 58.85) and GPT-4o (60.05 unified, 0.14 lower relation score) lead among LVLMs, while human captions score 62.99.
- The unified metric Sunified reaches Pearson 0.989, Kendall 0.927 and Spearman 0.982 with human judgment, versus at most 0.867 Pearson for BLEU@4 among traditional metrics.
- InternVL-Chat-V1-5 achieves the best CompreQA-P (91.67%) and CompreQA-Cap (94.33%) accuracy among LVLMs, with humans at 96.49% and 96.83%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the directed-scene-graph-based unified caption evaluation instead of direct Llama3 scoring | CompreCap caption evaluation · Human consistency across LVLMs | Kendall's τ: 0.309 -> 0.927 | Helped |
| Use the directed-scene-graph-based unified caption evaluation instead of direct Llama3 scoring | CompreCap caption evaluation · Human consistency across LVLMs | Spearman's ρ: 0.445 -> 0.982 | Helped |
| Generate captions with GPT-4o rather than LLaVA-Next-34B | CompreCap · CompreCap | unified score: 58.85 -> 60.05 | Helped |
| Generate captions with GPT-4o rather than LLaVA-Next-34B | CompreCap · CompreCap | object coverage (%): 72.86 -> 72.78 | Hurt |
| Use human-generated captions rather than GPT-4o captions | CompreCap · CompreCap | unified score: 60.05 -> 62.99 | Helped |
| Use InternVL-Chat-V1-5 rather than LLaVA-Next-34B | LVLM fine-grained object VQA · CompreQA-P | accuracy (%): 91.43±0.22 -> 91.67±0.00 | Helped |
| Use human responses rather than InternVL-Chat-V1-5 | Fine-grained object VQA · CompreQA-Cap | accuracy (%): 94.33±0.00 -> 96.83 | Helped |
Surprising findings
- Human captions scored by Llama3 directly are lower than most LVLM captions, showing that direct LLM scoring is unreliable without structured annotations.
- GPT-4V falls short of LLaVA-Next-34B on comprehensive captioning despite being a stronger model overall.
Limitations
- Benchmark is small (560 images) and relies on manual annotation, which is costly to scale.
- Evaluation depends on an LLM judge (Llama3) and noun-extraction pipeline, introducing stochastic variation (results averaged over three runs).
Experimental setup
Datasets: MSCOCO panoptic segmentation RAM vocabulary OpenImagesV4 vocabulary Objects365 vocabulary
Benchmarks: CompreCap CompreQA-P CompreQA-Cap DetailCaps-100
Metrics: object-level coverage attribute score relation score pixel coverage (S-Cov) unified score Pearson correlation Kendall tau Spearman rho
Baselines: BLEU@4 METEOR ROUGE-L CIDEr CLIPScore Llama3 direct scoring DetailCaps-100
Builds on: DetailCaps Factual scene graph parser Sentence-BERT spaCy
Closest papers by meaning
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.91)
- Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.90)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.89)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.89)
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.89)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.