Research map / Vision-language and multimodal / Multimodal LLMs
THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes THRONE, an object-based benchmark that uses an ensemble of open-source FLAN-T5 models to judge Type I (free-form) object hallucinations in LVLM descriptions on COCO and Objects365, and shows a simple object-enumeration data augmentation reduces them.
Problem
Existing hallucination benchmarks either target yes/no or multiple-choice questions (Type II) or rely on exact text matching (CHAIR) that cannot judge free-form LVLM descriptions (Type I), and POPE undersamples negatives; many also depend on closed API models.
Approach
Query LVLMs with a neutral "Describe this image in detail" prompt, then use an ensemble of open-source FLAN-T5 LMs and three question phrasings to answer abstractive yes/no existence questions for every COCO class, combined by unanimous voting with an ignore label; also complete POPE over all classes (POPE-C).
Key results
- On COCO, LLaVA-Mistral achieves the best class-wise F0.5 of 77.5 among 11 LVLMs, with class-wise precision 84.4; LLaVA-v1.5 scores 66.8.
- Spearman rank correlation between THRONE and POPE across 11 LVLMs is only 0.2, indicating Type I and Type II hallucination performance are largely disconnected.
- Adding object enumeration augmentation to LLaVA-v1.5 raises THRONE class-wise precision from 69.9 to 87.2 and F0.5 from 66.8 to 84.9 (COCO training).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add COCO object enumeration data during instruction tuning and inference | LLaVA-v1.5 · COCO | F^0.5_CLS: 66.8 -> 84.9 | Helped |
| Add COCO object enumeration data during instruction tuning and inference | LLaVA-v1.3 · COCO | F^0.5_CLS: 76.5 -> 79.9 | Helped |
| Add COCO and VisualGenome object enumeration data | LLaVA-v1.3 · COCO | F^0.5_CLS: 76.5 -> 81.5 | Helped |
| Add COCO and VisualGenome object enumeration data | LLaVA-v1.5 · COCO | F^0.5_CLS: 66.8 -> 84.1 | Helped |
| Add COCO object enumeration data | LLaVA-v1.5 · POPE | F1: 86.1 -> 87.0 | Helped |
| Add COCO object enumeration data | LLaVA-v1.3 · POPE-C | F1: 14.3 -> 17.2 | Helped |
| Evaluate using the POPE image and class subsampling strategy instead of all classes (ablation) | THRONE · COCO | F^0.5_CLS: 12.3 points | Hurt |
| Use simple majority voting (k = 5) instead of unanimous voting (k = 9) (ablation) | THRONE · COCO | compute metrics correlation: >0.99 | No clear effect |
| Use all-but-one voting (k = 8) instead of unanimous voting (k = 9) (ablation) | THRONE · COCO | compute metrics correlation: >0.99 | No clear effect |
| Replace THRONE's language-model judgments with human judgments (ablation) | THRONE · COCO subset | precision: 96% versus 91% | Helped |
Surprising findings
- Follow-up LVLM versions often show higher POPE scores but slightly lower THRONE scores, suggesting Type I and Type II hallucination reduction can be orthogonal.
- Completing POPE to all classes (POPE-C) drastically reduces precision and F1 for every evaluated model, showing POPE severely underestimates Type II hallucinations.
Limitations
- Judgement errors arise from COCO class definitions (e.g., tv includes monitors) and LM misjudgements.
- Unanimous voting marks some (response, class) pairs as ignored, reducing coverage.
- Evaluated models are limited to ~7B LLM LVLMs for fair comparison.
Experimental setup
Datasets: COCO 2017 validation Objects365 validation subset Visual Genome
Benchmarks: THRONE (COCO) THRONE (Objects365) POPE POPE-C
Metrics: class-wise precision class-wise recall F0.5 class-wise overall precision overall recall F1 Spearman rank correlation
Baselines: CHAIR POPE POPE-C MME MMBench
Builds on: CHAIR POPE LLaVA FLAN-T5
Closest papers by meaning
- Toward a Stable, Fair, and Comprehensive Evaluation of Object Hallucination in Large Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- Multi-Object Hallucination in Vision Language Models (NeurIPS 2024 · similarity 0.92)
- Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception (CVPR 2025 · similarity 0.92)
- Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback (AAAI 2025 · similarity 0.92)
- Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models (ICML 2024 · similarity 0.91)
- Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.