Research map / Vision-language and multimodal / Multimodal LLMs

ZINA: Multimodal Fine-grained Hallucination Detection and Editing

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes multimodal fine-grained hallucination detection and editing for MLLM captions via a two-stage detector-reviewer system (Zina), a new VisionHall dataset, and graph-based synthetic error injection.

Problem

MLLM image captions contain diverse hallucinations, but existing detectors are coarse-grained (sentence or long-span level) and do not provide span-level types or refinements suitable for editing.

Approach

Decouple token copying from detection: a detector MLLM finds hallucinated spans and types, a deterministic function inserts tags, and a reviewer MLLM validates tags and suggests edits; training data comes from error injection over dependency DAGs built with o3-mini.

Key results

• On VisionHall, Zina reaches 45.15 detection F1, 15.8 points above the best baseline GPT-4o (29.37).
• Zina achieves 44.02 BERT-F1 and 50.39 CLIP-F1 on editing, versus 24.89 and 30.19 for GPT-4o.
• On MHaluBench Image-to-Text, Zina improves claim-level Micro-F1 to 87.15 from 85.83 for GPT-based UniHD.

Surprising findings

• Converting baselines to the two-stage pipeline degraded GPT-4o and LLaVA-NeXT-Qwen-32B performance, while some models such as Qwen2.5-VL-72B improved.
• Sentence-level metrics CLIP-S and PAC-S differ by under 1 point across methods, motivating the proposed span-level metrics.

Limitations

• Two-stage architecture has relatively high inference time, hindering real-time deployment.
• The method often underdetects hallucinations because the reviewer MLLM is overly conservative.
• Evaluation relies on human-written reference captions (DCI), and the editing training data is partly synthetic.

Experimental setup

Datasets: VisionHall DCI MHaluBench HalLoc

Benchmarks: VisionHall MHaluBench HalLoc

Metrics: F1 CLIP-S PAC-S BERT-F1 CLIP-F1 claim-level F1 segment-level F1

Baselines: GPT-4o Llama-3.2-90B-Vision-Instruct Qwen2.5-VL-72B-Instruct LLaVA-OV-Qwen2-72B LLaVA-NeXT-Qwen-32B Qwen2-VL-7B LLaVA-1.5-7B UniHD Self-Check

Builds on: FAVA UniHD SelfCheckGPT HalLocalizer LLaMA-Factory

Closest papers by meaning

• Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning (NeurIPS 2025 · similarity 0.93)
• Detecting and Preventing Hallucinations in Large Vision Language Models (AAAI 2024 · similarity 0.91)
• Scalable Token-Level Hallucination Detection in Large Language Models (NeurIPS 2026 · similarity 0.91)
• Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinations (CVPR 2026 · similarity 0.90)
• FREAK: A Fine-grained Hallucination Evaluation Benchmark for Advanced MLLMs (ICLR 2026 · similarity 0.90)
• ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models (AAAI 2025 · similarity 0.90)