Research map / Large language models / Reasoning and chain of thought

Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UnifiedReward-Think, a unified multimodal reward model that learns long chain-of-thought reward reasoning via cold start distillation from GPT-4o, rejection sampling, and GRPO reinforcement fine-tuning.

Problem

Existing multimodal reward models for vision understanding and generation give direct answers or shallow reasoning, leading to inaccurate reward signals; scarce CoT reward data makes SFT-based long-CoT training hard.

Approach

Activate latent long-CoT reasoning in a unified image/video reward model (UnifiedReward) via a three-stage pipeline: GPT-4o distilled cold start on image-generation preferences, rejection sampling on correct unified-task reasoning, and GRPO with format and accuracy verifiable rewards on incorrectly predicted samples, plus multidimensional scoring for reasoning-decision consistency.

Key results

• On VLRewardBench image understanding, UnifiedReward-Think reaches 73.8 overall accuracy vs 67.5 for UnifiedReward (Hallucination subset 72.7 vs 58.1).
• On GenAI-Bench video generation (diff accuracy), the method reaches 82.3 vs 77.2 for UnifiedReward.
• Without explicit CoT at inference, the model still reaches 73.1 overall accuracy on VLRewardBench, above UnifiedReward's 67.5.

Measured effects reported in the paper

Surprising findings

• Cold start alone decreases VLRewardBench overall accuracy (67.5 to 66.9), with gains appearing only after rejection sampling and GRPO.
• Distilling CoT from Qwen2.5-VL-72B instead of GPT-4o yields comparable final performance.

Limitations

• Explicit long CoT increases inference time.
• Method is not optimized for tie cases; tau metrics are weaker than baselines trained with ties.
• Reinforcement learning may only amplify capabilities acquired during SFT, limiting gains beyond the base model.

Experimental setup

Datasets: HPD OIP EvalMuse OpenAI-4o_t2i_human_preference VideoDPO Text2Video-Human Preferences LLaVA-Critic-113K ShareGPTVideo-DPO

Benchmarks: VLRewardBench ShareGPTVideo GenAI-Bench VideoGen-RewardBench

Metrics: overall accuracy macro accuracy tau accuracy diff accuracy

Baselines: PickScore HPSv2 ImageReward VisionReward UnifiedReward VideoScore LiFT-Critic VideoReward LLaVA-Critic Gemini-1.5-Pro GPT-4o

Builds on: UnifiedReward GRPO DeepSeek-R1-Zero LLaVA-OneVision

Closest papers by meaning

• Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards (NeurIPS 2025 · similarity 0.91)
• Let VLMs Grade Their Own Thoughts: A Self-Quantification Approach to Reasoning-Aware Reward Modeling (CVPR 2026 · similarity 0.91)
• Latent Chain-of-Thought for Visual Reasoning (NeurIPS 2025 · similarity 0.91)
• Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modeling (NeurIPS 2026 · similarity 0.90)
• Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models (NeurIPS 2025 · similarity 0.89)
• RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning (ICLR 2026 · similarity 0.89)