Research map / Vision-language and multimodal / Multimodal LLMs

Continual SFT Matches Multimodal RLHF with Negative Supervision

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Argues that multimodal RLHF's advantage comes from negative supervision on rejected responses, and proposes nSFT, which adds an SFT loss on LLM-constructed corrective conversations built from rejected responses, matching or beating DPO/PPO-style alignment with one model.

Problem

Multimodal RLHF (DPO, PPO) is widely assumed superior to continual SFT for aligning VLMs to reduce hallucination, but it is memory-heavy (2-4 large models) and unstable; the paper asks whether it is truly superior and what makes it work.

Approach

DPO's gradient decomposes into a combination of chosen and rejected SFT gradients, so the missing piece in continual SFT is negative supervision. nSFT uses an LLM (GPT-4) guided by a vision error codebook to turn errors in rejected responses into corrective QA conversations, then trains them with a plain SFT loss alongside the GT captions.

Key results

• On LLaVA-150k 10k alignment data with LLaVA-1.5-7B, nSFT reaches +2.3 total improvement vs +0.5 for GT-DPO and -1.8 for continual SFT across nine benchmarks, with CHAIR reduced from 32.0 to 25.4.
• In the SOTA comparison (15k mixed data, LLaVA-1.5-7B), nSFT achieves MMBench 67.1 and MME 1531, outperforming SeVa-7B (MMBench 65.6, MME 1450) and SIMA-7B.
• With LLaVA-1.5-13B, nSFT reaches MME 1551 and POPE 88.6 vs 1520 and 87.5 for DPO on a 10k LLaVA-150k subset.

Measured effects reported in the paper

Surprising findings

• Naive continual SFT on the same data often hurts performance (e.g., -1.8 total on LLaVA-150k and -2.0 on TextCaps), contrary to the common assumption that SFT is merely inferior to RLHF.
• Removing the chosen (GT) response from nSFT leaves results nearly unchanged, and OCRVQA training without GT conversation is unstable.

Limitations

• Negative construction relies on GPT-4 to identify errors, and the paper states it is unclear how the approach transfers to NLP LLM alignment.
• Evaluation of MMHal used ChatGPT-3.5 instead of GPT-4 for quota reasons, and GPT-3.5 scores differ from GPT-4 scores.
• Gains saturate beyond about 15k-20k alignment data, attributed to data diversity overlap with LLaVA-1.5 SFT data.

Experimental setup

Datasets: OCRVQA TextCaps LLaVA-150k LLaVA 665k MS-COCO

Benchmarks: SQA GQA TextVQA MM-Vet MME MMBench POPE CHAIR MMHal-Bench SEED-Image MMBench-CN

Metrics: accuracy total improvement over baseline CHAIR MME score MM-Vet score instruction following score detail accuracy score

Baselines: GT-DPO SeVa SIMA Cont. SFT PPO CSR BLIP-2 InstructBLIP IDEFICS

Builds on: DPO PPO LLaVA-1.5 SeVa CSR

Closest papers by meaning

• RL makes MLLMs see better than SFT (ICLR 2026 · similarity 0.87)
• Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
• MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.87)
• Synthetic Data is an Elegant GIFT for Continual Vision-Language Models (CVPR 2025 · similarity 0.87)
• Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping (ICLR 2025 · similarity 0.87)
• CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.87)