Research map / Reinforcement learning / RL from feedback
Breaking the Self-Confirming Loop: Diagnosing and Mitigating Systemic Reward Bias in Self-Rewarding RL
Chuyi Tan, Peiwen Yuan, Xinglin Wang, Yiwei Li, Shaoxiong Feng, Yueqi Zhang, Jiayi Shi, Ji Zhang, Boyuan Pan, Yao Hu, Kan Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ensemble of sub-policies (k=2) instead of single-policy rewarding (ablation) | RLER (Interpolation v3) · Arithmetic test set (500 problems) | Avg@16: 63.2 -> 69.6 | Improved |
| add adaptive reward interpolation to ensemble SC (ablation) | RLER · Arithmetic test set (500 problems) | Avg@16: 67.6 -> 69.6 | Improved |
| add disagreement-aware rollout selection (ablation) | RLER · Arithmetic test set (500 problems) | Avg@16: 69.6 -> 71.5 | Improved |
| replace fixed alpha=0.5 with full adaptive alpha(x) in RLER (ablation) | RLER · DAPO-Math-17K, six benchmarks | Avg@8: 36.2 -> 37.5 | Improved |
| use full adaptive alpha(x) instead of annealed schedule (ablation) | single-policy interpolation · DAPO-Math-17K, six benchmarks | Avg@8: 33.6 -> 36.0 | Improved |
| increase ensemble size from k=2 to k=4 at fixed budget 16 (ablation) | RLER · DAPO-Math-17K, six benchmarks | Avg@8: 37.5 -> 37.6 | No clear effect |
| increase ensemble size from k=4 to k=8 at fixed budget 16 (ablation) | RLER · DAPO-Math-17K, six benchmarks | Avg@8: 37.6 -> 37.2 | Hurt |
| raise total rollout budget from 16 to 32 with k=4 (ablation) | RLER · DAPO-Math-17K, six benchmarks | Avg@8: 37.6 -> 37.9 | Improved |
| vary interpolation bounds beta- and beta+ (ablation) | RLER · DAPO-Math-17K, six benchmarks | Avg@8: 36.9 to 37.5 | No clear effect |
| use RLER instead of SC at temperature 0.5 | Llama-3.2-3B-Instruct arithmetic · Arithmetic test set | checkpoint accuracy variance: 78.92 -> 4.26 | Improved |
| use RLER instead of SC at temperature 0.9 | Qwen2.5-Math-7B · DAPO-Math-17K | Pass@1: 33.0 -> 37.5 | Improved |
| use disagreement-aware selection instead of selecting all rollouts when ensemble majority is wrong (ablation) | RLER · DAPO-Math-17K | reward noise rho_noise: 65.5 -> 50.5 | Improved |
Closest papers by meaning
- LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.92)
- Learning to Reason without External Rewards (ICLR 2026 · similarity 0.91)
- From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation (ICLR 2026 · similarity 0.91)
- Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models (ICLR 2026 · similarity 0.91)
- Self-Aligned Reward: Towards Effective and Efficient Reasoners (ICLR 2026 · similarity 0.90)
- The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
