Research map / Vision-language and multimodal / Multimodal LLMs
Toward a Stable, Fair, and Comprehensive Evaluation of Object Hallucination in Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies how instructions shape object hallucination in LVLMs via description length, then proposes LeHaCE, a length-hallucination curve framework that evaluates CHAIR at uniform lengths and reports the curve slope as a growth-rate metric.
Problem
Average-based object hallucination evaluation over instruction sets gives unstable and unfair comparisons because instruction sets induce image descriptions of very different lengths, and hallucination rises with length.
Approach
Hallucination degree (CHAIR) is found to be primarily driven by description length, with instructions acting indirectly through length. The authors fit a linear length-hallucination curve across instructions and evaluate at a fixed length (20/40/60/80 words), with the slope as a second metric measuring how fast hallucination grows with length.
Key results
- Pearson correlation between hallucination rate and average description length exceeds 0.6 for all 12 LVLMs, with 5 exceeding 0.9.
- LeHaCE gives lower RSD than the average-based framework for CHAIRI with 5+ instructions across all twelve LVLMs on MSCOCO, and for CHAIRS with 4+ instructions.
- On MSCOCO at 20 words InstructBLIP has the lowest LCI (2.35), while Gemini-Pro-Vision has the lowest growth rate in CHAIRI (0.05).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase image-description length | LVLM-generated image descriptions · MSCOCO images under different instructions | CHAIRI | Hurt |
| Increase image-description length | LVLM-generated image descriptions · MSCOCO images under different instructions | CHAIRS | Hurt |
| Use LeHaCE to evaluate at a uniform description length instead of averaging scores across instructions | Object hallucination evaluation · MSCOCO and NoCaps | RSD of CHAIR: 0.29 -> 0.11 | Helped |
| Use LeHaCE to evaluate at a uniform description length instead of averaging scores across instructions | Object hallucination evaluation · MSCOCO and NoCaps | RSD of CHAIR: 0.16 -> 0.04 | Helped |
| Use linear rather than polynomial fitting for the length-hallucination curve | LeHaCE · MSCOCO | RSD of CHAIRS: 0.05 -> 0.04 | Helped |
| Use linear rather than polynomial fitting for the length-hallucination curve | LeHaCE · MSCOCO | RSD of CHAIRS: 0.05; 0.04; 0.05 | Mixed |
Surprising findings
- Instructions affect hallucination mainly indirectly through description length, with a clear linear length-hallucination relation.
- InstructBLIP has the lowest hallucination at short lengths but the highest growth rate, leading to high hallucination on long descriptions.
Limitations
- Focuses only on object hallucination, not other hallucination types.
- Evaluates only subsets of each dataset (256 images) due to computational constraints.
- Only one proprietary LVLM (Gemini-Pro-Vision) evaluated due to API fees; linear fit unreliable with only 3 instructions.
Experimental setup
Datasets: MSCOCO NoCaps
Benchmarks: MSCOCO NoCaps
Metrics: CHAIRI CHAIRS LeHaCE (length-specific hallucination) LeHaCE growth rate (slope) relative standard deviation
Baselines: average-based evaluation framework (CHAIR averaged over instructions) polynomial (quadratic and cubic) curve fitting
Builds on: CHAIR POPE InstructBLIP instruction set
Closest papers by meaning
- Multi-Object Hallucination in Vision Language Models (NeurIPS 2024 · similarity 0.94)
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (ICLR 2024 · similarity 0.93)
- Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.93)
- Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.92)
- THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models (CVPR 2024 · similarity 0.92)
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.