Research map / Reinforcement learning / RL from feedback
Policy Filtration for RLHF to Mitigate Noise in Reward Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PF-PPO, which samples N responses, ranks them by a reward model, and trains PPO on a rank-filtered subset (best-random or best-worst) chosen by an R² reliability metric, improving code generation and math reasoning.
Problem
RLHF with PPO is hurt by an inaccurate reward model; the reward is reliable only in some reward regions (high or low), and moderate-reward samples provide noisy training signal, especially on complex reasoning tasks like code generation.
Approach
Filter out samples whose rewards are unreliable by sampling N responses, ranking them by reward, and training PPO only on a rank-based subset (best-random or best-worst); select the filtering strategy by the R² between rewards and actual scores on filtered samples, which correlates with final performance.
Key results
- PF-PPO (BR) reaches 82.9 pass@1 on HumanEval and 33.0 on LeetCode, versus 78.1 and 25.2 for PPO-S, with BW best on MBPP (76.2).
- On math, PF-PPO with ORM improves Ape210K from 84.1 to 86.2 and CMATH from 92.3 to 95.1 over PPO-S.
- R² on SFT policy correlates with downstream performance across filtering strategies; BoN filtering lowers R² (0.454) and performs worst (75.8 HumanEval).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PF-PPO (BR) instead of BOND | Code generation with deepseek-6.7B · HumanEval | pass@1: 80.8 -> 82.9 | Helped |
| Use PF-PPO (BW) instead of BOND | Code generation with deepseek-6.7B · MBPP | pass@1: 75.2 -> 76.2 | Helped |
| Use PF-PPO (BR) instead of BOND | Code generation with deepseek-6.7B · LeetCode Contest | pass@1: 30.0 -> 33.0 | Helped |
| Use PF-PPO (BR) instead of PPO-S with ORM | Math reasoning with Qwen1.5-7B · Ape210K | accuracy: 84.1 -> 86.2 | Helped |
| Use PF-PPO (BR) instead of PPO-S with ORM | Math reasoning with Qwen1.5-7B · CMATH | accuracy: 92.3 -> 95.1 | Helped |
| Use PF-PPO (BR) instead of PPO-S with Oracle | Math reasoning with Qwen1.5-7B · Ape210K | accuracy: 82.1 -> 83.8 | Helped |
| Use PF-PPO (BR) instead of PPO-S with Oracle | Math reasoning with Qwen1.5-7B · CMATH | accuracy: 90.8 -> 91.2 | Helped |
| Use PF-PPO (BR) instead of PPO-S with CRM | Math reasoning with Qwen1.5-7B · Ape210K | accuracy: 83.9 -> 84.3 | Helped |
| Use PF-PPO (BR) instead of PPO-S with CRM | Math reasoning with Qwen1.5-7B · CMATH | accuracy: 93.1 -> 94.2 | Helped |
| Replace rank-based BR/BW filtering with best-of-N filtering (ablation) | PF-PPO on code generation · HumanEval | pass@1: 82.9 -> 75.8 | Hurt |
| Use best-of-N filtering rather than unfiltered PPO (ablation) | PF-PPO on code generation · MBPP | pass@1: 73.8 -> 71.7 | Hurt |
| Use best-of-N filtering rather than unfiltered PPO (ablation) | PF-PPO on code generation · LeetCode Contest | pass@1: 25.2 -> 16.8 | Hurt |
Surprising findings
- Best-of-N filtering used in PPO hurts performance (75.8 on HumanEval), below the unfiltered PPO-S baseline, while BoN works well as an SFT target in BOND.
- Oracle (ground-truth) reward underperforms learned reward models on math tasks.
Limitations
- R² is an imperfect predictor for choosing a filtering strategy; the best strategy may vary by task, reward model, and base model.
- Experiments are limited to code and elementary-school math; the reliability pattern is shown only on a few benchmarks.
- Hyperparameter and threshold tuning for threshold-based baselines is coarse.
Experimental setup
Datasets: Magicoder-OSS-instruct evol-codealpaca-v1 LeetCode training problems Ape210K HumanEval MBPP CMATH
Benchmarks: HumanEval MBPP LeetCode Contest Ape210K CMATH multi-domain Doubao-25k tasks
Metrics: pass@1 R-squared best-of-1 accuracy best-of-5 accuracy
Baselines: SFT RAFT BOND DPO IPO KTO Iterative-DPO PPO-S PPO-M Magicoder
Builds on: PPO RLHF best-of-N sampling Bradley-Terry reward model OpenRLHF
Closest papers by meaning
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.91)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.91)
- DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.91)
- Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization (ICML 2024 · similarity 0.91)
- Greedy Sampling Is Provably Efficient For RLHF (NeurIPS 2025 · similarity 0.91)
- Discriminative Policy Optimization for Token-Level Reward Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.