Research map / Large language models / Reasoning and chain of thought
Discriminative Policy Optimization for Token-Level Reward Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Q-RM, a token-level reward model obtained by optimizing a discriminative policy whose logits act as token Q-values, trained from preference data and used to drive PPO/REINFORCE policy optimization.
Problem
Token-level process reward models built from DPO implicit rewards entangle reward modeling with language generation, causing unstable and inaccurate token credit assignment when training policies with RL.
Approach
Decouple reward modeling from generation: define a discriminative policy whose per-token logits Z(s_t,a_t) equal the optimal Q-function up to a state-dependent constant, so token rewards come from logits trained with a Bradley-Terry objective on preference pairs; advantages then follow from the logits without GAE.
Key results
- PPO+Q-RM improves average Pass@1 by 5.85 points over PPO+ORM and 4.56 over PPO+DPO-RM across GSM8K and MATH; REINFORCE+Q-RM reaches 93.48 Pass@16 on GSM8K and 62.87 on MATH.
- Q-RM converges about 12x faster than ORM on GSM8K and about 11x faster than step-level PRM on MATH.
- REINFORCE+Q-RM achieves 29.34% win rate and 27.07% LC win rate on AlpacaEval 2.0, above DPO (25.08/24.98).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Q-RM instead of ORM with PPO | PPO policy optimization · GSM8K and MATH | average Pass@1: +5.85 points | Helped |
| use Q-RM instead of ORM with REINFORCE | REINFORCE policy optimization · GSM8K and MATH | average Pass@1: +4.70 points | Helped |
| use Q-RM instead of DPO-RM with PPO | PPO policy optimization · GSM8K and MATH | average Pass@1: +4.56 points | Helped |
| use Q-RM instead of DPO-RM with REINFORCE | REINFORCE policy optimization · GSM8K and MATH | average Pass@1: +5.73 points | Helped |
| use Q-RM instead of DPO-RM with PPO | PPO policy optimization · QA-Feedback test | average reward: 0.5631±0.06 to 0.5956±0.04 | Helped |
| use Q-RM instead of DPO-RM with REINFORCE | REINFORCE policy optimization · QA-Feedback test | average reward: 0.5581±0.06 to 0.5917±0.07 | Helped |
| use Q-RM instead of ORM with PPO (ablation) | PPO policy optimization · GSM8K training | sample efficiency: nearly 12× | Helped |
| use Q-RM instead of step-level PRM with PPO (ablation) | PPO policy optimization · MATH training | sample efficiency: nearly 11× | Helped |
| use Q-RM instead of step-level PRM with PPO (ablation) | PPO policy optimization · MATH test | Pass@1: 28.20% to 31.60% | Helped |
| increase the number of samples N (ablation) | REINFORCE policies trained with different reward models · GSM8K test | Pass@N | Mixed |
| use a larger policy model (ablation) | PPO+Q-RM · GSM8K | Pass@1 | Helped |
| use Q-RM rather than ORM or DPO-RM (ablation) | REINFORCE policy optimization with matched reward-model and policy-model sizes · GSM8K and MATH test | Pass@1 | Helped |
Surprising findings
- Q-RM has the lowest pairwise accuracy on RewardBench and in GSM8K RM accuracy, yet yields the best policy optimization performance.
- Higher reward model accuracy does not translate to better RL policy performance.
Limitations
- Relies on initialization from strong large LLMs; performance tied to backbone strength.
- Value function approximated as Z*(s_t,a_t) in PPO, an inconsistency that may hurt the critic; relative BT rewards used as absolute returns.
- Evaluation focuses on policy training rather than inference-time use; gamma treated as constant approximation.
Experimental setup
Datasets: GSM8K MATH QA-Feedback UltraFeedback PRM800K RewardBench
Benchmarks: GSM8K MATH QA-Feedback AlpacaEval 2.0 RewardBench
Metrics: Pass@1 Pass@8 Pass@16 reward on relevance/factuality/completeness RMs win rate LC win rate pairwise accuracy
Baselines: DPO-RM Implicit-PRM ABC-RM PGG-RM ORM DPO SimPO ORPO RLOO GRPO RFT
Builds on: DPO PPO REINFORCE Bradley-Terry model maximum entropy RL GAE
Closest papers by meaning
- RM-R1: Reward Modeling as Reasoning (ICLR 2026 · similarity 0.90)
- Q-Probe: A Lightweight Approach to Reward Maximization for Language Models (ICML 2024 · similarity 0.90)
- Policy Filtration for RLHF to Mitigate Noise in Reward Models (ICML 2025 · similarity 0.90)
- Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.90)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.90)
- Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Models (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.