Research map / Reinforcement learning / RL from feedback
REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge
Yasi Zhang, Tianyu Chen, Mingyuan Zhou, Oscar Leong, Ying Nian Wu, Michal Lukasik
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace binary-accuracy RL with REAL regression-aware policy-dependent reward | RLOO RL on Mistral2-7B · Judge benchmarks average | Pearson: 64.1 -> 67.9 | Improved |
| use regression-aware but policy-independent reward (Standard RL(reg)) instead of accuracy reward (ablation) | Standard RL on Mistral2-7B · Judge benchmarks average | Pearson: 64.1 -> 63.4 | No clear effect |
| add RL with REAL on top of TRACT SFT checkpoint | TRACT on Mistral2-7B · Judge benchmarks average | Pearson: 63.2 -> 67.9 | Improved |
| add RL with REAL on top of RAFT SFT checkpoint | RAFT on Qwen3-32B · Judge benchmarks average | Pearson: 62.6 -> 71.0 | Improved |
| add RL with REAL on top of RAFT SFT checkpoint | RAFT on Qwen3-8B · Judge benchmarks average | Pearson: 61.9 -> 67.0 | Improved |
| increase group size K from 4 to 16 (ablation) | REAL · Judge benchmarks average | Pearson: 63.7 -> 67.9 | Improved |
| increase group size K from 8 to 16 (ablation) | REAL · Judge benchmarks average | Pearson: 67.2 -> 67.9 | Improved |
| add log-likelihood term (lambda=1) to regression reward (ablation) | REAL · Judge benchmarks average | Pearson: 66.0 -> 67.9 | Improved |
| raise lambda from 1 to 10 (ablation) | REAL · Judge benchmarks average | Pearson: 67.9 -> 68.1 | No clear effect |
| restrict training to partially correct prompts (0<acc<1) rather than all prompts (ablation) | REAL (K=8) · Judge benchmarks average | Pearson: 65.5 -> 67.2 | Improved |
| add prediction weight beta=0.01 versus beta=0 (ablation) | REAL init from SFT · Judge benchmarks average | Pearson: 67.3 -> 67.9 | Improved |
| raise prediction weight beta from 0.01 to 1 (ablation) | REAL init from SFT · Judge benchmarks average | Pearson: 67.9 -> 66.7 | Hurt |
Closest papers by meaning
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.92)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)
- TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.90)
- Tricks or Traps? A Deep Dive into RL for LLM Reasoning (ICLR 2026 · similarity 0.90)
- The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
