AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization

Shiping Gao, Hongzhan Chen, Xiaojun Quan, Qifan Wang, Lifu Huang

ICML 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train IPVRM on prefix values instead of sequence-level log-ratio (Implicit PRM)implicit PRM · ProcessBench avg F1F1: 25.4 -> 39.1 (Qwen3-0.6B); 27.9 -> 34.0 (Llama3.2-1B)Improved
train IPVRM instead of Implicit PRMimplicit PRM · Best-of-N MATH-500 and Minerva-Mathavg accuracy: 25.1 -> 28.1 (Qwen3-0.6B); 46.7 -> 46.8 (Qwen3-8B)Mixed
add distribution-level candidate-token TD updates (DistRL) to GRPO with IPVRM (ablation)GRPO w/ IPVRM · AIME/MATH500/Minerva/Olympiad/AMC avg@8avg@8 accuracy: 16.3 -> 16.7 (Qwen3-0.6B); 36.3 -> 37.7 (Qwen3-8B)Improved
add DistRL to GRPO with DPO-RM (ablation)GRPO w/ DPO-RM · avg@8, Qwen3-0.6Bavg@8 accuracy: 14.0 -> 14.1No clear effect
add DistRL to GRPO with Implicit PRM (ablation)GRPO w/ Implicit PRM · avg@8, Qwen3-0.6Bavg@8 accuracy: 13.3 -> 14.7Improved
swap DPO-RM for IPVRM as reward model in DistRL (ablation)DistRL · avg@8, Qwen3-0.6Bavg@8 accuracy: 14.1 -> 16.7Improved
upweight later prefixes in IPVRM loss (ablation)IPVRM · Best-of-N MATH-500, Qwen3-0.6Bavg accuracy: 45.2 -> 46.5Improved
upweight later prefixes in IPVRM loss (ablation)IPVRM · ProcessBench, Qwen3-0.6Bavg F1: 39.1 -> 37.2Hurt
upweight earlier prefixes in IPVRM loss (ablation)IPVRM · ProcessBench, Qwen3-0.6Bavg F1: 39.1 -> 40.1Improved
upweight earlier prefixes in IPVRM loss (ablation)IPVRM · Best-of-N MATH-500, Qwen3-0.6Bavg accuracy: 45.2 -> 43.1Hurt
add naive online RM updates without ADB or DLW (ablation)IPVRM online update · Llama3.2-1B RLRM accuracy / policy rewardHurt
add ADB and DLW balancing to online IPVRM updates (ablation)IPVRM online update · avg@8, Qwen3-0.6B with DistRLavg@8 accuracy: 15.4 -> 16.7Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy