Research map / Reinforcement learning / RL from feedback
Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization
Shiping Gao, Hongzhan Chen, Xiaojun Quan, Qifan Wang, Lifu Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train IPVRM on prefix values instead of sequence-level log-ratio (Implicit PRM) | implicit PRM · ProcessBench avg F1 | F1: 25.4 -> 39.1 (Qwen3-0.6B); 27.9 -> 34.0 (Llama3.2-1B) | Improved |
| train IPVRM instead of Implicit PRM | implicit PRM · Best-of-N MATH-500 and Minerva-Math | avg accuracy: 25.1 -> 28.1 (Qwen3-0.6B); 46.7 -> 46.8 (Qwen3-8B) | Mixed |
| add distribution-level candidate-token TD updates (DistRL) to GRPO with IPVRM (ablation) | GRPO w/ IPVRM · AIME/MATH500/Minerva/Olympiad/AMC avg@8 | avg@8 accuracy: 16.3 -> 16.7 (Qwen3-0.6B); 36.3 -> 37.7 (Qwen3-8B) | Improved |
| add DistRL to GRPO with DPO-RM (ablation) | GRPO w/ DPO-RM · avg@8, Qwen3-0.6B | avg@8 accuracy: 14.0 -> 14.1 | No clear effect |
| add DistRL to GRPO with Implicit PRM (ablation) | GRPO w/ Implicit PRM · avg@8, Qwen3-0.6B | avg@8 accuracy: 13.3 -> 14.7 | Improved |
| swap DPO-RM for IPVRM as reward model in DistRL (ablation) | DistRL · avg@8, Qwen3-0.6B | avg@8 accuracy: 14.1 -> 16.7 | Improved |
| upweight later prefixes in IPVRM loss (ablation) | IPVRM · Best-of-N MATH-500, Qwen3-0.6B | avg accuracy: 45.2 -> 46.5 | Improved |
| upweight later prefixes in IPVRM loss (ablation) | IPVRM · ProcessBench, Qwen3-0.6B | avg F1: 39.1 -> 37.2 | Hurt |
| upweight earlier prefixes in IPVRM loss (ablation) | IPVRM · ProcessBench, Qwen3-0.6B | avg F1: 39.1 -> 40.1 | Improved |
| upweight earlier prefixes in IPVRM loss (ablation) | IPVRM · Best-of-N MATH-500, Qwen3-0.6B | avg accuracy: 45.2 -> 43.1 | Hurt |
| add naive online RM updates without ADB or DLW (ablation) | IPVRM online update · Llama3.2-1B RL | RM accuracy / policy reward | Hurt |
| add ADB and DLW balancing to online IPVRM updates (ablation) | IPVRM online update · avg@8, Qwen3-0.6B with DistRL | avg@8 accuracy: 15.4 -> 16.7 | Improved |
Closest papers by meaning
- Free Process Rewards without Process Labels (ICML 2025 · similarity 0.91)
- Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning (ICLR 2025 · similarity 0.91)
- Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.90)
- From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling (AAAI 2026 · similarity 0.89)
- ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs (NeurIPS 2025 · similarity 0.89)
- Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
