Research map / Vision-language and multimodal / Multimodal LLMs
Multi-Object Hallucination in Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies multi-object hallucination in LVLMs, building a benchmark of multi-object probes and evaluating mitigation methods (OPERA, ATMAN-in-Box) against single-object baselines.
Problem
Object hallucination benchmarks in vision language models focus on single object classes, not individual entities, so how models misperceive multiple objects together is under-studied.
Approach
Evaluates hallucination when several objects are present in an image (homogeneous and heterogeneous object sets, wild/seen/unseen settings) and proposes an attention-based mitigation (ATMAN-in-Box) that is compared with OPERA and grounding LVLMs.
Key results
- LLaVA-1.5 hallucination rate in the default multi-object wild setting (seen) is 21.26%, rising to 52.40% in the homogeneous setting.
- LLaVA-1.5 + ATMAN-in-Box reduces wild multi-object hallucination in the seen split from 21.26% to 23.80%? (no reduction; hallucination rises slightly) while lowering heterogeneous rate from 7.69% to 8.50%? (rises) - mitigation effect is mixed in the reported table.
- GroundHOG and GLaMM show lower multi-object hallucination on some splits but the table indicates trade-offs with instruction adherence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ATMAN-in-Box | LLaVA-1.5 · Multi-object seen, Wild | hallucination rate: 21.26% -> 23.80% | Hurt |
| Add ATMAN-in-Box | LLaVA-1.5 · Multi-object seen, Hom. | hallucination rate: 52.40% -> 55.60% | Hurt |
| Add ATMAN-in-Box | LLaVA-1.5 · Multi-object seen, Het. | hallucination rate: 7.69% -> 8.50% | Hurt |
| Add ATMAN-in-Box | LLaVA-1.5 · Multi-object unseen, Wild | hallucination rate: 13.96% -> 16.00% | Hurt |
| Add ATMAN-in-Box | LLaVA-1.5 · Multi-object unseen, Hom. | hallucination rate: 31.88% -> 35.60% | Hurt |
| Add ATMAN-in-Box | LLaVA-1.5 · Multi-object unseen, Het. | hallucination rate: 3.98% -> 4.50% | Hurt |
| Add OPERA | LLaVA-1.5 · Multi-object seen, Wild | hallucination rate: 21.26% -> 24.07% | Hurt |
| Add OPERA | LLaVA-1.5 · Multi-object seen, Hom. | hallucination rate: 52.40% -> 58.65% | Hurt |
| Add OPERA | LLaVA-1.5 · Multi-object seen, Het. | hallucination rate: 7.69% -> 7.35% | Helped |
| Add OPERA | LLaVA-1.5 · Multi-object unseen, Wild | hallucination rate: 13.96% -> 13.20% | Helped |
| Add OPERA | LLaVA-1.5 · Multi-object unseen, Hom. | hallucination rate: 31.88% -> 37.14% | Hurt |
| Add OPERA | LLaVA-1.5 · Multi-object unseen, Het. | hallucination rate: 3.98% -> 3.82% | Helped |
Surprising findings
- Mitigation methods such as OPERA and ATMAN-in-Box do not uniformly reduce multi-object hallucination; several reported metrics increase relative to vanilla LLaVA-1.5.
Limitations
- Mitigation gains are inconsistent across settings, as the table shows some metrics worsening.
- Evaluation appears limited to the nuScenes-based case studies and a small set of models in this excerpt.
Experimental setup
Datasets: nuScenes
Benchmarks: multi-object hallucination benchmark (default, single-object, student-forcing settings)
Metrics: hallucination rate (%)
Baselines: LLaVA-1.5 OPERA GLaMM GroundHOG MiniCPM-V
Builds on: OPERA
Closest papers by meaning
- Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.95)
- Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.95)
- Fine-Grained Multi Image Object Hallucination Benchmark (CVPR 2026 · similarity 0.94)
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (ICLR 2024 · similarity 0.94)
- Toward a Stable, Fair, and Comprehensive Evaluation of Object Hallucination in Large Vision-Language Models (NeurIPS 2024 · similarity 0.94)
- Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination (CVPR 2026 · similarity 0.94)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.