Research map / Vision-language and multimodal / Multimodal LLMs

GHOST: Hallucination-Inducing Image Generation for Multimodal LLMs

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GHOST, which optimizes a CLIP image embedding via an MLP mapper into MLLM vision space, then conditions Stable Diffusion unCLIP to generate natural-looking, object-free images that induce object hallucinations in MLLMs.

Problem

Object hallucination in MLLMs is studied with static benchmarks and fixed images, which cannot reveal model-specific or unanticipated hallucination vulnerabilities; an automatic way to actively generate hallucination-inducing images is needed.

Approach

Decouple the attack from the generator: learn an MLP mapper from CLIP embeddings to the MLLM's vision encoder space, optimize the CLIP embedding to raise the MLLM's "Yes" probability while penalizing similarity to the target object's text, and then decode with unCLIP starting from a partially noised latent of the original image, filtered by OWLv2.

Key results

• GHOST yields 2,816 hallucination-inducing images for Qwen2.5-VL (about 29% success), versus 99 found by DASH-LLM and DASH-OPT combined on COCO.
• Images optimized on Qwen2.5-VL induce hallucinations in GPT-4o at a 66.5% rate.
• Fine-tuning Qwen2.5-VL with LoRA on GHOST images raises POPE from 88.7 to 93.2 and lowers CHAIRs from 3.8 to 2.9.

Surprising findings

• Human annotators judged 86-89% of GHOST images as object-free, while prompting the MLLM to say yes, indicating subtle semantic cues rather than visible objects.
• Reasoning model GLM-4.1V-Thinking was hallucinated by optimizing only the first token after the think tag, without optimizing the final answer directly.

Limitations

• DASH comparison is on COCO rather than its original ReLAION-5B setting, and DASH-OPT could not run at batch size 1 on 80GB GPUs in the authors' reproduction.
• GLM-generated images show higher FID, since the optimization targets a thinking token rather than the final answer.
• Mitigation experiment is a small toy setup with a single fine-tuning run.

Experimental setup

Datasets: COCO ObjectNet

Benchmarks: POPE CHAIR VQAv2 COCO Captioning

Metrics: hallucination success rate FID SSIM human yes-rate accuracy

Baselines: DASH-LLM DASH-OPT Stable Diffusion v2.1 Stable Diffusion unCLIP

Builds on: DASH Stable Diffusion unCLIP OWLv2 LoRA

Closest papers by meaning

• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination (CVPR 2026 · similarity 0.93)
• Fine-Grained Multi Image Object Hallucination Benchmark (CVPR 2026 · similarity 0.92)
• Analyzing and Mitigating Object Hallucination: A Training Bias Perspective (AAAI 2026 · similarity 0.92)
• Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splitting (NeurIPS 2026 · similarity 0.92)
• FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMs (ICLR 2026 · similarity 0.92)
• Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance (ICML 2025 · similarity 0.92)