Research map / Vision-language and multimodal / Multimodal LLMs

Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Preference tuning against hallucination · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UE-DPO, a DPO variant that uses token-level epistemic uncertainty (logit gap between blurred-image and clean-image predictions) to reweight token gradients in preferred and dispreferred responses for hallucination mitigation in MLLMs.

Problem

DPO-based hallucination mitigation for MLLMs assigns token-level credit using self-assessed visual sensitivity, which biases training toward already well-learned visual cues and neglects hard-to-perceive critical details.

Approach

Replace visual-sensitivity credit signals with an epistemic uncertainty measure (logit of blurred-image top prediction minus logit of ground-truth token on clean image); intensify gradients on visually insensitive, high-uncertainty tokens in preferred responses and attenuate penalties on visually sensitive, high-uncertainty tokens in dispreferred responses, with a theoretical link to an entropy-re

Key results

• On LLaVA-v1.5-7B trained on RLAIF-V data, UE-DPO reaches MMHal-Bench score 2.95 with HalRate 0.37, and CHAIRs 11.62 on Object HalBench.
• On Qwen2.5-VL-3B with RLHF-V 5.7k pairs, UE-DPO improves MMHal score from 2.53 (DPO) to 3.05 and lowers CHAIRs from 30.6 to 16.7 (DPO's value); AMBER F1 89.5.
• On LLaVA-v1.5-7B with RLHF-V data, UE-DPO attains MMHal score 2.82 versus 2.47 for TPO and 2.16 for V-DPO, and AMBER-d F1 85.7 versus 85.0 for TPO's reported F1 on AMBER-d.

Surprising findings

• AMBER-d accuracy drops slightly relative to baselines (e.g., 79.1 vs 79.3 for TPO on LLaVA-7B RLHF-V data), which the authors attribute to the model becoming more cautious.
• Using the preferred branch alone yields most of the gains, while the dispreferred penalty-attenuation branch alone is weak.

Limitations

• Object HalBench CHAIRi is much worse than the RLAIF-V baseline on LLaVA-7B RLHF-V data (6.69 vs 3.70), and CHAIRs trails RLAIF-V on some settings.
• AMBER-d accuracy is not consistently best, and gains are sensitive to dataset scale and coverage.
• Epistemic uncertainty can be confounded by perception bottlenecks on tiny objects, as seen in the visualization.

Experimental setup

Datasets: RLHF-V RLAIF-V

Benchmarks: Object HalBench MMHal-Bench AMBER

Metrics: CHAIRs CHAIRi MMHal score hallucination rate AMBER CHAIR AMBER Cog accuracy F1

Baselines: POVID mDPO HALVA RLHF-V RLAIF-V V-DPO TPO LLaVA-RLHF

Builds on: DPO V-DPO TPO

Closest papers by meaning

• CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.93)
• Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization (NeurIPS 2025 · similarity 0.93)
• P$^2$-DPO:Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization (ICLR 2026 · similarity 0.92)
• Automated Multi-level Preference for MLLMs (NeurIPS 2024 · similarity 0.92)
• Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs (ICML 2026 · similarity 0.92)
• Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization (ICML 2026 · similarity 0.91)