Research map / Vision-language and multimodal / Multimodal LLMs

Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds RLAIF-V-Bias, a preference dataset of language-biased and vision-biased negatives created by masking modalities in LLaVA-v1.5-7B, and proposes NaPO, a noise-aware DPO variant using a negative Box-Cox (MAE/BCE) loss with margin-based dynamic robustness, to reduce modality bias and hallucination.

Problem

MLLMs over-rely on a single modality (language priors or visual details), producing biased and hallucinated responses; automatically constructed debiasing preference data is noisy, which harms standard DPO.

Approach

Generate biased negative responses by masking visual or textual inputs to induce modality reliance, and replace the DPO BCE loss with a negative Box-Cox loss (interpolating MAE and BCE) whose noise-robustness coefficient q is set dynamically from the reward margin of each sample.

Key results

• On LLaVA-v1.5-7B, the RLAIF-V-Bias loss gets CB 58.9 and LP 44.0 on VLind-Bench versus 39.4 and 25.4 for DPO on RLAIF-V, and CHAIRs drops from 32.0 to 25.7.
• The paper reports an average ~19% improvement on the modality bias benchmark over the strongest baseline.
• On LLaVA-v1.5-13B, the method gives CB 42.1 and LP 25.1 on VLind-Bench, improving over DPO on RLAIF-V (37.1, 21.2).

Measured effects reported in the paper

Surprising findings

• Replacing DPO with NaPO in the full objective degrades performance (LP 21.1 vs 44.0), and dynamic weight balancing can cause collapse on the original RLAIF-V data.
• Standard LMDPO (length-normalized DPO) on RLAIF-V collapses the commonsense bias score to 0.3 on VLind-Bench.

Limitations

• Noise robustness is only validated on noise from automatically constructed data by LLaVA-v1.5-7B, not on broader LLM-synthesized noise.
• Hyperparameter alpha must be tuned per data type and margin definition (sum vs average logP), and inappropriate choices degrade performance.
• Whether bias is always harmful is left unresolved.

Experimental setup

Datasets: RLAIF-V

Benchmarks: VLind-Bench Object HalBench AMBER MMHalBench

Metrics: Language Prior accuracy Commonsense Bias accuracy CHAIRs CHAIRi AMBER CHAIR/coverage/HalRate/Cog MMHalBench score hallucination rate

Baselines: DPO MDPO GPT-4V POVID OPERA VCD EOS HA-DPO HALVA RLHF-V HSA-DPO RLAIF-V

Builds on: DPO RLAIF-V Generalized Cross Entropy MDPO

Closest papers by meaning

• CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.90)
• Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• Automated Multi-level Preference for MLLMs (NeurIPS 2024 · similarity 0.88)
• Explore How to Inject Beneficial Noise in MLLMs (AAAI 2026 · similarity 0.88)
• MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.88)
• Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.88)