Research map / Vision-language and multimodal / Multimodal LLMs

Dynamics-Aware Preference Optimization for Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CW-DPO, a two-stage VLM preference optimization method (constrained SFT with gentle negatives, then DPO with a competence-aware cooling weight on the loser term) to mitigate a "squeezing effect" from easy negatives.

Problem

Preference-based finetuning of vision-language models is unstable: easy or trivially wrong negatives produce large, misaligned gradients that distort optimization, reduce diversity and degrade calibration.

Approach

Stage 1 applies a Lagrangian-relaxed constrained SFT (ReLU penalty keeping negative NLL above a threshold) to flatten the loss landscape; Stage 2 scales the loser term of the DPO loss by a sigmoid cooling weight based on the average per-token log-probability of the loser, down-weighting easy negatives and keeping hard on-policy contrasts.

Key results

• On COCO Test, CW-DPO reaches CIDEr 142.6, +3.4 over PPO (139.2), with BLEU-4 39.6.
• On MMMU, CW-DPO achieves 74.6% accuracy versus 73.1% for OPA-DPO; MMBench1.1 89.6%.
• Compared with vanilla DPO, CW-DPO shows lower TV/JS divergence (~0.15/0.10 vs ~0.45/0.30) and ECE stabilized at 0.08-0.10 vs rising to 0.25.

Surprising findings

• Vanilla DPO underperforms SFT on lexical metrics such as BLEU-4 on COCO.
• Removing Smooth SFT reduces COCO CIDEr by about 5 points.

Limitations

• Requires paired preference data with reliable positive-negative supervision.
• Cooling weight introduces hyperparameters (tau, l_floor) that may need per-dataset tuning.
• Evaluated only on captioning-style VLMs and benchmarks; not validated on video or embodied or long-horizon reasoning.

Experimental setup

Datasets: COCO Flickr30k NoCaps MMMU MMBench

Benchmarks: COCO Flickr30k NoCaps MMMU MMBench1.1

Metrics: BLEU-4 METEOR CIDEr SPICE accuracy ECE total variation Jensen-Shannon divergence entropy

Baselines: SFT DPO PPO V-DPO GRPO OPA-DPO

Builds on: DPO PPO GRPO V-DPO OPA-DPO

Closest papers by meaning

• Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search (NeurIPS 2026 · similarity 0.89)
• SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes (NeurIPS 2025 · similarity 0.88)
• Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs (NeurIPS 2025 · similarity 0.88)
• Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment (ICLR 2025 · similarity 0.88)
• Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.88)