Research map / Vision-language and multimodal / Multimodal LLMs
Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AllPath, a training-free multi-path attention-head intervention for LVLM hallucination that jointly targets text-to-text and image-to-text heads, selected via LPI-based and attention-ratio scores, and shows gains across POPE, MCQ-POPE, CHAIR, and MME.
Problem
LVLMs hallucinate, but existing intervention methods target only a single causal path (image-to-output or text-to-text), so they fail to generalize across benchmarks with different question-answer alignment formats.
Approach
Decomposes hallucination into image-to-input-text, image-to-output-text, and text-to-text paths; identifies T2T heads via a refined log-probability-increase score and I2T heads via attention ratio to object-aligned image regions in a single forward pass; then scales selected heads (suppress T2T promoters, amplify T2T and I2T suppressors of hallucination) with format-adaptive selection.
Key results
- On LLaVA-v1.5-7B POPE COCO random, AllPath reaches 87.2 accuracy / 86.0 F1 versus 85.1 / 83.7 for vanilla, and on MCQ-POPE random 80.5 accuracy vs 72.8.
- On CHAIR, AllPath lowers CHAIR_S from 52.2 (vanilla) to 26.6 and CHAIR_I from 14.6 to 7.2, comparable to PAI's 28.8/7.9.
- On MME hallucination subsets, AllPath scores 600.0 total versus 540.0 for vanilla on LLaVA-v1.5-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply AllPath multi-path head intervention instead of vanilla decoding | LLaVA-1.5-7B · POPE COCO | F1: 83.7 -> 86.0 | Helped |
| apply AllPath multi-path head intervention instead of vanilla decoding | LLaVA-1.5-7B · MCQ-POPE COCO | Macro F1: 72.9 -> 80.5 | Helped |
| apply AllPath multi-path head intervention instead of vanilla decoding | LLaVA-1.5-7B · CHAIR | C_S: 52.2 -> 26.6 | Helped |
| apply AllPath multi-path head intervention instead of vanilla decoding | LLaVA-1.5-7B · MME hallucination subset | Total: 540.0 { (±39.69)} -> 600.0 { (±8.66)} | Helped |
| add I2T head intervention to T2T heads (ablation) | AllPath · POPE COCO Random | F1: 84.9 -> 86.0 | Helped |
| add T2T head intervention to I2T heads (ablation) | AllPath · POPE COCO Random | F1: 85.2 -> 86.0 | Helped |
| add both T2T and I2T head interventions (ablation) | LLaVA-1.5-7B · POPE COCO Random | F1: 83.7 -> 86.0 | Helped |
| vary ξ from 20 to 10 (ablation) | AllPath · POPE COCO Random | F1: 86.0 -> 86.2 | Helped |
| vary ξ from 20 to 30 (ablation) | AllPath · POPE COCO Random | F1: 86.0 -> 87.7 | Helped |
| vary ξ from 20 to 40 (ablation) | AllPath · POPE COCO Random | F1: 86.0 -> 87.9 | Helped |
| vary ζ from 10 to 5 (ablation) | AllPath · POPE COCO Random | F1: 86.0 -> 84.8 | Hurt |
| vary ζ from 10 to 15 (ablation) | AllPath · POPE COCO Random | F1: 86.0 -> 87.4 | Helped |
Surprising findings
- Knocking out output-to-image attention barely hurts POPE (about 2%) but substantially worsens CHAIR (about 10%), indicating different pathways dominate different question formats.
- T2T head rankings correlate strongly within the same alignment format (rho about 0.82) but drop to about 0.12 across formats such as Yes/No vs MCQ.
Limitations
- Does not investigate which causal pathways are most effective for different question types.
- Does not explore training strategies to improve pathway use.
- Limited by the scope of existing hallucination benchmarks, which may miss subtle hallucinations.
Experimental setup
Datasets: POPE COCO A-OKVQA GQA
Benchmarks: POPE MCQ-POPE CHAIR MME
Metrics: accuracy F1 CHAIR_S CHAIR_I MME hallucination score
Baselines: Vanilla VCD ICD PAI AD-HH
Builds on: Log Probability Increase PAI AD-HH VCD
Closest papers by meaning
- Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering (ICLR 2025 · similarity 0.93)
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs (AAAI 2026 · similarity 0.93)
- Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models (ICLR 2025 · similarity 0.93)
- Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models (CVPR 2026 · similarity 0.92)
- Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.92)
- Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating (ICML 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.