Research map / Vision-language and multimodal / Multimodal LLMs
PhD: A ChatGPT-Prompted Visual Hallucination Evaluation Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds PhD, a ChatGPT-assisted semi-automated visual hallucination benchmark (14.6k daily images, 750 counter-common-sense images, 102k binary VQA triplets) across five recognition tasks and four modes, and evaluates 15 open-source and 3 proprietary MLLMs.
Problem
Existing objective visual hallucination evaluation (VHE) benchmarks are small, select hallucinatory items without image-specific reasoning, and do not explicitly link evaluation to the causes of MLLM hallucination, and are saturating as models improve.
Approach
Structure a binary VQA benchmark along task and mode dimensions, using ChatGPT to select task- and image-specific hallucinatory items, generate hitem-embedded questions, specious/incorrect contexts, and counter-common-sense images with Text2Image tools, mapping each mode to a cause of hallucination.
Key results
- LLaVA-OneVision achieves the best open-source PhD Index of 0.698, followed by Molmo (0.690) and InternVL-1.5 (0.561).
- GPT-4o reaches a PhD Index of 0.812 on random-2k, well above the best open-source model, despite similar POPE and AMBER scores.
- LLaVA-1.5-L (13B) and LLaVA-1.5 (7B) score nearly identically overall (0.270 vs 0.265), with the larger LLM helping in PhD-sec and PhD-icc but hurting PhD-base.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a larger LLM (Vicuna-7B-1.5 to Vicuna-13B-1.5) in LLaVA-1.5 (ablation) | LLaVA-1.5 · PhD | PhD Index: 0.265 -> 0.270 | No clear effect |
| Use a larger LLM (Vicuna-7B-1.5 to Vicuna-13B-1.5) in LLaVA-1.5 (ablation) | LLaVA-1.5 · PhD-base | PhD-base Index: 0.443 -> 0.422 | Hurt |
| Use a larger LLM (Vicuna-7B-1.5 to Vicuna-13B-1.5) in LLaVA-1.5 (ablation) | LLaVA-1.5 · PhD-sec | PhD-sec Index: 0.082 -> 0.099 | Helped |
| Use a larger LLM (Vicuna-7B-1.5 to Vicuna-13B-1.5) in LLaVA-1.5 (ablation) | LLaVA-1.5 · PhD-icc | PhD-icc Index: 0.011 -> 0.019 | Helped |
| Use a larger LLM (Vicuna-7B-1.5 to Vicuna-13B-1.5) in LLaVA-1.5 (ablation) | LLaVA-1.5 · PhD-ccs | PhD-ccs Index: 0.534 -> 0.542 | Helped |
| Use a larger LLM in InstructBLIP (Vicuna-7B to 13B) (ablation) | InstructBLIP-L · PhD-base | PhD-base Index: 0.535 -> 0.324 | Hurt |
| Apply VCD decoding mitigation to Qwen-VL | Qwen-VL · PhD (full-set) | PhD Index: 0.488 -> 0.560 | Helped |
| Apply VCD decoding mitigation to LLaVA-1.6-L | LLaVA-1.6-L · PhD (full-set) | PhD Index: 0.423 -> 0.511 | Helped |
| Higher-resolution / stronger visual encoder inputs (visual input strength) | open-source MLLMs · PhD-base | PhD-base Index | Helped |
| Using PhD-icc/PhD-sec context (textual inconsistency) instead of base questions | open-source MLLMs · PhD-icc and PhD-sec | PhD Index | Hurt |
| Woodpecker mitigation on LLaVA-1.6-L | LLaVA-1.6-L · PhD (random-2k) | PhD Index | No clear effect |
Surprising findings
- A larger LLM does not necessarily reduce hallucination; LLaVA-1.5-L is nearly tied with the 7B variant overall.
- Open-source MLLMs have a strong say-yes bias (average 0.611), and PhD Index correlates strongly negatively with say-yes rate.
Limitations
- Relies on MS-COCO/TDIUC images that may have been seen during MLLM training, an intentional data leakage choice.
- Manual inspection is still needed to verify ChatGPT-generated hitems, contexts and CCS images.
- Proprietary model evaluations are restricted to a random 2k subset due to budget.
Experimental setup
Datasets: TDIUC MS-COCO
Benchmarks: PhD POPE AMBER
Metrics: PhD Index (harmonic mean of Yes and No recall) say-yes rate accuracy
Baselines: POPE AMBER ROME NOPE CIEM VCD Woodpecker
Builds on: TDIUC POPE AMBER ROME CLIP
Closest papers by meaning
- Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.92)
- Detecting and Preventing Hallucinations in Large Vision Language Models (AAAI 2024 · similarity 0.92)
- Seeing Is Believing: Rich-Context Hallucination Detection for MLLMs via Backward Visual Grounding (AAAI 2026 · similarity 0.91)
- MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM (NeurIPS 2025 · similarity 0.91)
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models (ICLR 2024 · similarity 0.91)
- Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.