Research map / Vision-language and multimodal / Multimodal LLMs

RL makes MLLMs see better than SFT

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Compares SFT and DPO post-training of MLLMs and their vision encoders, finds DPO yields stronger MLLM and vision-encoder performance, and proposes PIVOT, which applies DPO-based LLM-head training to existing vision encoders to improve them for MLLM use.

Problem

Whether RL (DPO) versus SFT post-training differently affects MLLM performance and, critically, the vision encoder's visual representations, which prior work largely overlooked in favor of LLM-centric explanations.

Approach

Controlled comparison of SFT and DPO on the same image-query-response pairs within the LLaVA-OneVision pipeline, then analysis of the vision encoder via ImageNet linear probing, segmentation probing, gradient visualization and representational alignment; the PIVOT recipe trains existing vision encoders with an LLM head using DPO and reuses them in MLLMs.

Key results

• DPO beats SFT on vision-centric and OCR&Chart tasks with SigLIP2-g/16 (+4.2 and +2.4 points for SigLIP2-L/16), while Knowledge VQA gains are marginal.
• PIVOT-enhanced SigLIP2-So/16 reaches 55.6 average MLLM score versus 53.9 for SigLIP2-g/16, despite 2.5x fewer parameters.
• PIVOT lifts classical encoders such as CLIP (46.3 to 49.5) and MAE (36.8 to 39.7) in MLLM average score.

Surprising findings

• A vision encoder trained with DPO inside an MLLM outperforms the SFT version even on ImageNet linear probing, suggesting RL reshapes visual representations.
• An encoder trained with an SFT-only objective on ImageNet also benefits from PIVOT, indicating gains are not limited to self-supervised encoders.

Limitations

• Controlled comparison uses a single preference dataset (MPO) that may favor DPO; SFT-friendly filtered data narrows but does not reverse the gap.
• SFT is not equalized with negative examples, so supervision signals differ between SFT and DPO.
• Evaluated mostly with a small-scale protocol (Qwen2.5-1.5B head, 20K preference pairs).

Experimental setup

Datasets: LLaVA-OneVision-3.2M MPO LAION/CC/SBU-558K Cambrian 737K ImageNet ADE20K MMPR

Benchmarks: Cambrian 16-task VQA suite ImageNet ADE20K HallusionBench POPE

Metrics: average VQA score ImageNet top-1 linear probe accuracy patch-level recall

Baselines: SFT MPO GRPO PPO SigLIP2-g/16

Builds on: LLaVA-OneVision DPO MPO SigLIP2 Cambrian

Closest papers by meaning

• Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.91)
• Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance (CVPR 2026 · similarity 0.89)
• Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation (NeurIPS 2025 · similarity 0.89)
• Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs (NeurIPS 2026 · similarity 0.89)
• Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders (ICLR 2026 · similarity 0.89)
• Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders (ICLR 2025 · similarity 0.88)