Research map / Reinforcement learning / RL from feedback

Beyond SFT-to-RL: Pre-alignment via Black-box On-policy Distillation for Multimodal RL

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PRISM, inserting a black-box adversarial on-policy distillation alignment stage with an MoE perception/reasoning discriminator between SFT and RLVR for Qwen3-VL, improving downstream GRPO/DAPO/GSPO results.

Problem

SFT on curated demonstrations induces distributional drift that degrades original capabilities and compounds during subsequent RLVR in large multimodal models, especially for perception versus reasoning.

Approach

Insert a logit-free, response-level adversarial on-policy distillation stage between SFT and RLVR, where a Mixture-of-Experts discriminator with separate visual-description and reasoning-trace experts (Bradley-Terry trained, jointly updated with the policy via GRPO) provides disentangled rewards toward the supervision distribution.

Key results

• PRISM+GRPO improves average accuracy over SFT→GRPO by +4.4 points on Qwen3-VL-4B (66.2 vs 61.8).
• PRISM+GRPO improves average accuracy over SFT→GRPO by +6.0 points on Qwen3-VL-8B (69.3 vs 63.3).
• Removing the alignment stage costs 4.4 avg points on 4B; removing SFT costs 16.8 avg points; a dense 4B discriminator costs 3.4 avg points.

Surprising findings

• The PRISM checkpoint before RLVR shows average accuracy (59.3 on 8B, 57.2 on 4B) comparable to or below the SFT checkpoint, so gains come only after RLVR.
• SFT degrades the Qwen3-VL Instruct checkpoint on average, and on 8B SFT→GRPO/GSPO barely recovers the original Instruct performance.

Limitations

• Alignment stage adds training cost and memory since the policy and MoE discriminator must be jointly maintained.
• MoE design relies on structured response formats (caption and reasoning trace), limiting applicability to other tasks.
• Distribution analysis uses structural proxies (reasoning steps, caption items) rather than direct distributional divergence measures.

Experimental setup

Datasets: Gemini 3 Flash distilled 113K multimodal reasoning corpus 1.26M public Gemini-family demonstrations MathVista MathVerse MathVision WeMath MMMU MMMU-Pro HallusionBench

Benchmarks: MathVista MathVerse MathVision WeMath MMMU MMMU-Pro HallusionBench

Metrics: accuracy

Baselines: Qwen3-VL Instruct SFT only SFT+GRPO SFT+DAPO SFT+GSPO

Builds on: GRPO DAPO GSPO GKD Bradley-Terry reward modeling

Closest papers by meaning

• Perception-Aware Policy Optimization for Multimodal Reasoning (ICLR 2026 · similarity 0.90)
• Revisual-R1: Advancing Multimodal Reasoning From Optimized Cold Start to Staged Reinforcement Learning (ICLR 2026 · similarity 0.90)
• R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoning (CVPR 2026 · similarity 0.89)
• Reinforcing Multimodal Reasoning Against Visual Degradation (NeurIPS 2026 · similarity 0.89)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoning (CVPR 2026 · similarity 0.89)
• Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.89)