Research map / Large language models / Reasoning and chain of thought

Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modeling

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Process reward models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DeScore, a video reward model that decouples an MLLM's chain-of-thought reasoning from a discriminative scoring head (learnable query token + regression head), trained with a discriminative cold start with random CoT masking followed by dual-objective GRPO + BT RL.

Problem

Video reward models face a dilemma: discriminative RMs regress rewards without reasoning and are prone to shortcut learning, while generative CoT RMs generalize better but suffer from unstable GRPO-based optimization when reasoning and scoring are coupled in one sampling chain.

Approach

Decouple reasoning from scoring: the MLLM first generates a CoT, then a learnable [Reward] token appended after the CoT is regressed to a scalar via a BT-trained head. Training uses a discriminative cold start with random CoT masking, then GRPO on the CoT plus an auxiliary BT loss on the scalar reward.

Key results

• DeScore reaches 0.734 in-domain accuracy versus 0.642 for the best discriminative baseline (VideoAlign) and 0.617 for VideoScore2.
• On VideoGen-Bench, DeScore achieves 0.768 Acc w/o Tie versus 0.722 for VideoAlign and 0.582 for UnifiedReward-Thinking.
• DeScore uses about 76% less training data than SOTA reward models while outperforming them, per the paper; with Longcat-GRPO on Wan-2.1-1.3B, Aesthetic Quality rises from 0.547 to 0.645 on VBench.

Surprising findings

• Training with GRPO alone on the RL stage drops VideoGen-Bench accuracy from 0.768 to 0.648, showing the reward head needs the auxiliary BT loss.
• Coupled generative CoT RM training shows pronounced accuracy fluctuations, while BT-based discriminative training converges smoothly.

Limitations

• Primarily evaluates text-video faithfulness, so may be less effective at detecting motion implausibility and visual artifacts.
• Relies on Gemini-2.5-Pro and Qwen3-VL-8B generated CoTs filtered by consistency with human labels.

Experimental setup

Datasets: in-domain preference dataset (22K pairs for training, 1469 for evaluation) GenAI VideoGen-Bench VBench

Benchmarks: GenAI VideoGen-Bench VBench

Metrics: preference accuracy w/ tie preference accuracy w/o tie

Baselines: VideoScore VideoAlign VisionReward UnifiedReward UnifiedReward-Thinking VideoScore2

Builds on: GRPO Bradley-Terry loss VideoAlign UnifiedReward-Thinking

Closest papers by meaning

• Let VLMs Grade Their Own Thoughts: A Self-Quantification Approach to Reasoning-Aware Reward Modeling (CVPR 2026 · similarity 0.91)
• Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.90)
• Latent Chain-of-Thought for Visual Reasoning (NeurIPS 2025 · similarity 0.88)
• Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.88)
• VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning (NeurIPS 2025 · similarity 0.88)
• VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models (CVPR 2025 · similarity 0.88)