Research map / Vision-language and multimodal / Multimodal LLMs

Fine-Grained Multi Image Object Hallucination Benchmark

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MIOH, a multi-image object hallucination benchmark crossing four object tasks with three reasoning patterns under three adversarial pressures, and evaluates 29 MLLMs.

Problem

Existing object hallucination benchmarks target single images with binary questions, and multi-image benchmarks lack controlled manipulation of visual difficulty and reasoning patterns to diagnose object hallucination.

Approach

Crosses four object-centric tasks (existence, counting, attribute, position) with three multi-image reasoning patterns (comprehensive, comparative, selective) and controls adversarial pressures (number of images, hard positives via occlusion/small size or CLIP filtering, hard negatives via co-occurrence bias or CLIP confusion), using MCQ format.

Key results

• Across 29 evaluated models, overall average accuracy is only 36.1% on MIOH.
• Gemini-2.5-Pro (64.4%) and GPT-5 (63.1%) lead; best open-source are Qwen2-VL-7B (49.1%) and MiniCPM-V-2.6 (48.5%).
• Counting is the hardest task at 25.4% average accuracy, and increasing images to 8 drops existence accuracy to 30.0%.

Surprising findings

• Open-source models with higher easy-question accuracy show larger robustness drops under adversarial pressure (e.g., Qwen2-VL-7B -27.2%).
• Single-image decomposed evaluation outperforms simultaneous multi-image comprehensive evaluation for the same content across models.

Limitations

• Counting questions limited to images with 5 or fewer objects due to annotation reliability, and annotation noise remains for position and counting.
• Final benchmark is a stratified subset (3,484 questions) of roughly 20,000 verified questions.
• Commercial model parameter sizes are not disclosed, so size-robustness correlations cover open-source models only.

Experimental setup

Datasets: COCO-ReM PACO Synthetic Visual Genome (SVG) COCO

Benchmarks: MIOH

Metrics: accuracy

Baselines: GPT-5 Gemini-2.5-Pro Qwen2-VL Qwen2.5-VL LLaVA-OneVision LLaVA-Interleave InternVL3.5 Mantis-8B MiniCPM-V-2.6 Ovis2.5 Phi-4-multimodal

Builds on: COCO-ReM PACO Synthetic Visual Genome Visual Haystack RePOPE

Closest papers by meaning

• Multi-Object Hallucination in Vision Language Models (NeurIPS 2024 · similarity 0.94)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.93)
• MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM (NeurIPS 2025 · similarity 0.93)
• GHOST: Hallucination-Inducing Image Generation for Multimodal LLMs (ICLR 2026 · similarity 0.92)
• Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models (ICLR 2026 · similarity 0.92)
• Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (ICLR 2024 · similarity 0.92)