Research map / Vision-language and multimodal / Multimodal LLMs
Adaptive Hallucination Alleviation in Multimodal Large Language Models: From Strategic Data Selection to Severity-Guided Training
Yuanyi Xu, Xiangru Zhu, Sihang Jiang, Zhixu Li, Bei Yang, Xiaoxiao Xu, Yanghua Xiao, Wei Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an adaptive framework for alleviating hallucination in multimodal LLMs that strategically selects hallucination-related training samples and applies severity-guided training to them.
Problem
MLLMs suffer from multiple forms of hallucination, but prior methods struggle to efficiently build high-quality hallucination-related samples and to process them at fine granularity.
Approach
Combines strategic data selection for constructing hallucination-related samples with a severity-guided training scheme that treats samples according to their hallucination severity rather than uniformly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add image selection (CLIP-S and PPL under noise) (ablation) | DPO · AMBER discriminative | accuracy: 77.6 -> 78.4 | Improved |
| add image selection (ablation) | DPO · HallusionBench | aAcc: 43.72 -> 44.29 | Improved |
| add explicit severity weighting in the DPO loss (ablation) | DPO · MMHal-Bench | score: 2.37 -> 2.51 | Improved |
| add explicit severity weighting in the DPO loss (ablation) | DPO · AMBER generative | CHAIR: 2.5 -> 2.1 | Improved |
| add severity-weighted visual attention reallocation (ablation) | HD-DPO (explicit) · MMHal-Bench | score: 2.51 -> 2.60 | Improved |
| add severity-weighted visual attention reallocation (ablation) | HD-DPO (explicit) · HallusionBench | aAcc: 44.64 -> 45.46 | Improved |
| add severity-weighted visual attention reallocation (ablation) | HD-DPO (explicit) · MMBench | score: 64.3 -> 64.0 | Hurt |
| train on Object-type hallucinations only (vs Secondary types) (ablation) | HD-DPO · AMBER/MMHal/CHAIR | CHAIR_S: 47.3 -> 36.7 (Object); Action 43.0 | Improved |
| apply HD-DPO to other MLLMs | Qwen-VL / InstructBLIP / InternLM-XComposer2 · AMBER, CHAIR | CHAIR_S: Qwen-VL 39.7 -> 10.3; InstructBLIP 21.0 -> 4.2 | Improved |
| apply HD-DPO instead of V-DPO | V-DPO · HallusionBench | aAcc: 46.20 -> 45.46 | Hurt |
| apply HD-DPO instead of CSR | CSR · MMBench | score: 64.1 -> 64.0 | Hurt |
| apply HD-DPO to LLaVA-1.5-7B | LLaVA-1.5-7B · AMBER discriminative | accuracy: 72.3 -> 79.7 | Improved |
Closest papers by meaning
- Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization (NeurIPS 2025 · similarity 0.94)
- Automated Multi-level Preference for MLLMs (NeurIPS 2024 · similarity 0.93)
- Stop Learning it all to Mitigate Visual Hallucination, Focus on the Hallucination Target. (CVPR 2025 · similarity 0.93)
- Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.93)
- CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.93)
- Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback (AAAI 2025 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
