Research map / Vision-language and multimodal / Multimodal LLMs

Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MISP-DPO, a multimodal DPO method that uses multiple semantically diverse negative images selected via CLIP embeddings and a sparse autoencoder, optimized with a Plackett-Luce objective and importance sampling.

Problem

Multimodal DPO for vision-language models typically uses a single oversimplified negative image (perturbed or retrieved), which captures only one error dimension and can cause optimization bias and hallucinations.

Approach

Selects several semantically diverse negative images from an open-domain pool by decomposing CLIP-space difference vectors with a sparse autoencoder (scoring by reconstruction error, deviation, and diversity), then trains with a multi-negative Plackett-Luce DPO loss using an importance-sampling estimator.

Key results

• On LLaVA-1.5-7B, MISP-DPO reaches +30.09% average improvement over base and reduces MMHalBench hallucination rate to 32.29% (base 51.04%).
• On Qwen2.5-VL-7B, MISP-DPO achieves +5.35% average improvement with POPE accuracy 88.66 vs 87.65 base.
• On Qwen2.5-VL-3B, MISP-DPO achieves +19.89% average improvement and WildVision reward 8.6 vs 7.5 for DPO.

Surprising findings

• Adding five negatives reduces HallusionBench performance, attributed to noisy redundant negatives.
• On Qwen2.5-VL-7B, DPO achieves the lowest MMHalBench hallucination rate among methods on that metric (11.46 tied with MISP-DPO), and base WildVision reward/winrate is higher than all tuned methods.

Limitations

• Evaluation relies on GPT-based scoring which may introduce bias, as noted by the authors.
• Gains on some benchmarks are small or negative (e.g., Qwen2.5-VL-7B POPE/MMVP and WildVision for base model), and results are from single runs with small benchmarks such as WildVision 500 samples.
• Training uses only about 5K preference samples and a single small LoRA configuration; baselines' settings differ (CHiP full fine-tuning).

Experimental setup

Datasets: RLHF-V-Dataset COCO train2017

Benchmarks: MMHalBench HallusionBench POPE WildVision MMVP

Metrics: score hallucination rate accuracy reward win rate average improvement over base

Baselines: DPO mDPO CHiP Random multi-negative Base (no preference tuning) diffusion negatives crop+diffusion negatives similarity-based negatives

Builds on: DPO mDPO CHiP Plackett-Luce multi-negative DPO CLIP

Closest papers by meaning

• MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models (ICLR 2025 · similarity 0.89)
• Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs (NeurIPS 2025 · similarity 0.89)
• CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.88)
• ISR-DPO: Aligning Large Multimodal Models for Videos by Iterative Self-Retrospective DPO (AAAI 2025 · similarity 0.88)
• Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.87)