Research map / Vision-language and multimodal / Multimodal LLMs

MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Decoding-time hallucination mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MoD-DPO (and MoD-DPO++ with language-prior debiasing), a KL-regularized DPO variant that enforces invariance to irrelevant-modality corruption and sensitivity to relevant-modality corruption, trained on an auto-generated 18.1k audiovisual preference dataset to reduce cross-modal hallucinations in omni LLMs.

Problem

Omni-modal LLMs hallucinate across modalities (e.g., hearing sounds from visual cues or seeing events prompted by audio) due to spurious inter-modality correlations and over-reliance on language priors.

Approach

Adds modality-aware KL regularizers to the DPO objective: invariance to corrupted irrelevant modality and sensitivity to corrupted relevant modality, derives a closed-form optimal policy and reward, and adds a language-prior debiasing penalty using a text-only reference log-probability.

Key results

• On AVHBench, MoD-DPO++ on Qwen 2.5 Omni reaches 88.19 accuracy on audio-driven video hallucination vs 84.15 for the base model, and 69.68 vs 54.69 on audiovisual matching.
• On CMM, MoD-DPO++ on Qwen 2.5 Omni achieves overall pa 89.2 and hr 87.2 (vs 86.4 and 84.6 base), exceeding OmniDPO (86.6/84.7).
• Ablation on Qwen 2.5 Omni: sensitivity+invariance raises AVHBench accuracy from 72.07 to 79.74; adding LPD gives 80.42.

Surprising findings

• Using diffusion-based noise corruption with 500 steps outperforms fewer steps, which appear too similar to the original input.
• Including joint audiovisual objective gave no significant improvement over audio and visual objectives alone.

Limitations

• Does not explicitly promote beneficial cross-modal synergy for tasks requiring complementary audiovisual reasoning.
• Relies on approximation that corrupted-input target distributions are fixed within each step.
• Preference data generated with GPT-4o and other automatic tools, with no human verification reported.

Experimental setup

Datasets: MSR-VTT VALOR32K AudioCaps AVHBench CMM DailyOmni MVBench MMAU OmniDialog

Benchmarks: AVHBench CMM DailyOmni MVBench MMAU OmniDialog

Metrics: accuracy precision recall F1 perception accuracy (pa) hallucination resistance (hr)

Baselines: DPO OmniDPO VideoLLaMA 2 VITA-1.5 OmniVinci Qwen 3 Omni VCD AVCD

Builds on: DPO OmniDPO Visual Contrastive Decoding LLaMAFactory

Closest papers by meaning

• The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio (NeurIPS 2025 · similarity 0.93)
• AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models (ICLR 2025 · similarity 0.93)
• CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.92)
• AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding (NeurIPS 2025 · similarity 0.92)
• Robust Multimodal Large Language Models Against Modality Conflict (ICML 2025 · similarity 0.91)
• Cross-Modal Attention Calibration for LVLM Hallucination Mitigation (CVPR 2026 · similarity 0.91)