Research map / Reinforcement learning / RL from feedback
Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QRPO, which regresses log-probability ratios onto quantile-transformed absolute rewards so the partition function has a closed form, enabling offline pointwise-reward policy fitting.
Problem
Policy fitting methods like DPO and REBEL rely on relative (preference or reward-difference) signals because the KL-regularized RL partition function is considered intractable for absolute rewards, which wastes absolute reward information from strong reward models and verifiable rewards.
Approach
Transform absolute rewards into quantiles of the reference reward distribution, which are uniform, so the partition function equals the MGF of the uniform distribution, beta(exp(1/beta)-1); this yields a closed-form pointwise squared-regression objective, with reference-sample pre-computation estimating the quantiles.
Key results
- On Llama 8B with Magpie-Air, QRPO reaches ArmoRM length-controlled reward 0.2005 vs 0.1975 for SimPO and 0.1904 for DPO.
- On Llama 8B with UltraFeedback, QRPO achieves 65.6% AlpacaEval 2 WR vs 39.4% for DPO and 39.5% for REBEL.
- QRPO shows less length bias than DPO and SimPO in the implicit-reward vs standardized-length analysis, with REBEL also showing no trend.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QRPO instead of preference-fitting baselines | Llama 8B Tülu 3 SFT · Magpie-Air test | ArmoRM reward (LC): 0.1975 -> 0.2005 | Helped |
| Use QRPO instead of preference-fitting baselines | Llama 8B Tülu 3 SFT · UltraFeedback test | AlpacaEval 2 win rate (%): 39.2 -> 65.6 | Helped |
| Use QRPO instead of preference-fitting baselines | Mistral 7B Instruct v0.2 · Magpie-Air test | ArmoRM reward (LC): 0.1898 -> 0.1893 | Hurt |
| Use QRPO instead of preference-fitting baselines | Mistral 7B Instruct v0.2 · UltraFeedback test | AlpacaEval 2 LC win rate (%): 38.8 -> 38.8 | No clear effect |
| Increase the number of reference rewards used to estimate quantile rewards (ablation) | QRPO · Magpie-Air off-policy | online test reward | Helped |
| Increase the number of reference rewards used to estimate quantile rewards (ablation) | QRPO · Magpie-Air offline | online test reward | No clear effect |
| Add online completions alongside offline completions during training (ablation) | QRPO on Magpie-Air · Magpie-Air | reward improvement: 26% greater reward improvement | Helped |
| Use a robust scalar reward directly rather than converting it to preferences | QRPO vs. DPO and SimPO · Magpie-Air length-bias analysis | length bias | Helped |
| Use SimPO length normalization instead of DPO | DPO policy · Magpie-Air length-bias analysis | length bias | No clear effect |
Surprising findings
- SimPO, despite reducing average length, still shows the same length-bias slope as DPO.
- In the off-policy regime, training samples' probabilities generally decrease overall for small beta, yet performance still improves.
Limitations
- Quantile rewards are defined relative to the reference policy and become less informative as the model improves beyond it.
- Transformation functions beyond the identity were not explored empirically.
- Experiments are offline only; the effect of online data is only briefly checked in an appendix ablation.
Experimental setup
Datasets: Magpie-Air UltraFeedback LeetCode
Benchmarks: AlpacaEval 2 LeetCode
Metrics: ArmoRM reward (original and length-controlled) AlpacaEval 2 length-controlled win rate AlpacaEval 2 win rate test-case pass rate
Baselines: DPO REBEL SimPO Initial SFT-chosen
Builds on: DPO REBEL SimPO KL-regularized RL objective
Closest papers by meaning
- Discriminative Policy Optimization for Token-Level Reward Models (ICML 2025 · similarity 0.89)
- GVPO: Group Variance Policy Optimization for Large Language Model Post-Training (NeurIPS 2025 · similarity 0.89)
- AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.88)
- DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.88)
- Policy Filtration for RLHF to Mitigate Noise in Reward Models (ICML 2025 · similarity 0.88)
- Q-Probe: A Lightweight Approach to Reward Maximization for Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.