Research map / Reinforcement learning / RL from feedback
Tournament Style RL: Stabilizing Policy Optimization on Non Verifiable Problems
Gurusha Juneja, Shubham Milind Phal, Jennifer She, Lisa Wang, Dorsa Sadigh, Anca Dragan, William Yang Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use win-rate against 4 fixed anchors instead of a single scalar rubric score as reward | GRPO on LLaMA-3.1-8B · Stories | win-rate (%): 47.5 -> 74.0 | Improved |
| use multiple anchors instead of a single anchor comparison (ablation) | TSRL · Stories | win-rate (%): 13.5 -> 74.0 | Improved |
| increase number of anchors k from 1 to 10 (ablation) | TSRL · all four tasks | win-rate: average of 52 win-rate points from k = 1 to k = 10 | Improved |
| use diverse anchors instead of low-diversity anchors (ablation) | TSRL · Stories (LLaMA-3.1-8B) | win-rate (%): 12.8 -> 74.0 | Improved |
| replace Kimi-K2 judge with a weak LLaMA-3.1-8B judge (ablation) | TSRL · Stories (LLaMA-3.1-8B) | win-rate (%): 74.0 -> 70.0 | Hurt |
| replace Kimi-K2 judge with a weak judge (ablation) | Rubric-based RL · Stories (LLaMA-3.1-8B) | win-rate (%): 47.5 -> 34.0 | Hurt |
| use higher-quality anchors (Flash -> Pro -> human) (ablation) | TSRL · Stories (LLaMA-3.1-8B) | win-rate (%): 65.5 -> 70.0 -> 74.0 | Improved |
| replace fixed anchors with in-batch ranking reward (ablation) | TSRL · Stories (LLaMA-3.1-8B) | win-rate (%): 74.0 -> 55.0 | Hurt |
| use online win-rate reward instead of offline DPO on the same anchor comparisons | LLaMA-3.1-8B · Stories | win-rate (%): 57.5 -> 74.0 | Improved |
| refresh anchors from the current policy every 50 steps (self-bootstrap) (ablation) | TSRL · Stories (LLaMA-3.1-8B) | win-rate (%): 74.0 -> 81.5 | Improved |
| scale policy size from Qwen2.5-3B to 7B (ablation) | TSRL · Stories | win-rate (%): gap over rubric 14.2 -> 28.0 | Improved |
| use ordered partial-correctness anchors instead of binary correctness reward | LLaMA-3.1-8B · MATH | accuracy: 50.0 -> 54.5 | Improved |
Closest papers by meaning
- Policy Filtration for RLHF to Mitigate Noise in Reward Models (ICML 2025 · similarity 0.89)
- Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning (ICLR 2025 · similarity 0.88)
- Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias (NeurIPS 2026 · similarity 0.87)
- Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR (NeurIPS 2026 · similarity 0.87)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.87)
- PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
