Research map / Reinforcement learning / RL from feedback
Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a symmetric RL loss (adding a reverse cross-entropy-style term to A2C and PPO) to improve robustness to advantage-sign noise, evaluated on Atari, MuJoCo/Box2D with reward noise, and PPO-based RLHF on IMDB and TL;DR.
Problem
RL training, especially A2C/PPO and RLHF with reward models, is unstable because advantage signs are noisy (moving targets, high gradient variance, reward model errors, off-policy effects), which resembles label noise in classification.
Approach
Adapts reverse cross-entropy from Symmetric Cross Entropy to policy gradients, defining a symmetric RL loss L = alpha*L_RL + beta*L_reverse (SA2C and SPPO) computed over discretized action distributions, so that wrong-sign advantages are counteracted.
Key results
- SPPO beats PPO on 16 of 22 Atari games without noise and 19 of 22 with 10% BSC reward noise.
- On TL;DR with GPT-J, SPPO raises win rate to 52.50% vs 43.25% for PPO and RM score to 6.13 vs 5.94.
- On IMDB, SPPO reaches RM score 0.92 vs 0.89 for PPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add reverse RL loss to PPO | GPT-2 · IMDB positive sentiment | RM Score: 0.89 $\pm$ 0.02 -> 0.92 $\pm$ 0.01 | Helped |
| Add reverse RL loss to PPO | GPT-2 · IMDB positive sentiment | Perplexity: 41.09 $\pm$ 0.43 -> 40.60 $\pm$ 0.44 | Helped |
| Add reverse RL loss to PPO | GPT-J · TL;DR summarization | RM Score: 5.94 $\pm$ 0.08 -> 6.13 $\pm$ 0.02 | Helped |
| Add reverse RL loss to PPO | GPT-J · TL;DR summarization | Win Rate: 43.25 $\pm$ 3.82 -> 52.50 $\pm$ 2.40 | Helped |
| Add reverse RL loss to PPO | GPT-J · TL;DR summarization | Perplexity: 19.08 $\pm$ 0.17 -> 19.27 $\pm$ 0.21 | Hurt |
| Add reverse RL loss to discretized PPO | DPPO · MuJoCo and Box2D with Gaussian reward noise | Mean final score | Mixed |
| Add reverse RL loss to PPO | PPO · Atari games without reward noise | Win count: 16 out of 22 wins | Mixed |
| Add reverse RL loss to PPO | PPO · Atari games with BSC reward noise | Win count: 19 out of 22 wins | Helped |
| Vary reverse-loss weight $\beta$ (ablation) | SPPO · Atari games with and without reward noise | Percentage improvement over PPO | Helped |
| Use $\beta=20.0$ rather than $\beta=0.2$ (ablation) | SPPO with Qwen2-0.5B · TL;DR summarization | Reward performance | Mixed |
Surprising findings
- SA2C gives only marginal gains over A2C, while SPPO is consistently better, attributed to PPO's off-policy updates and advantage normalization on small minibatches.
- Empty summaries can receive higher reward-model scores than valid summaries, showing reward model errors on out-of-distribution outputs.
Limitations
- Continuous-action methods require discretizing the action space to compute reverse terms over other actions.
- Hyperparameters beta and Z interact; sensitivity varies by task and SA2C gains are marginal and narrow in beta range.
- Qwen2-0.5B results show performance drops after many epochs for both PPO and SPPO.
Experimental setup
Datasets: Atari games MuJoCo Box2D IMDB TL;DR
Benchmarks: Atari games MuJoCo Box2D IMDB positive sentiment TL;DR summarization
Metrics: final episode score reward model score perplexity win rate
Baselines: A2C PPO DA2C DPPO SFT
Builds on: Symmetric Cross Entropy PPO A2C Stable-Baselines3 TRIL
Closest papers by meaning
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.88)
- Robust Reinforcement Learning from Corrupted Human Feedback (NeurIPS 2024 · similarity 0.88)
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (ICLR 2024 · similarity 0.88)
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (ICML 2024 · similarity 0.87)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.87)
- On the Robustness of Reward Models for Language Model Alignment (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.