Research map / Reinforcement learning / RL from feedback
Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes A-LoL, a family of advantage-weighted offline policy gradient methods that treat whole LM outputs as single actions and train only on positive-advantage pre-existing data, compared against PPO, DPO, PRO, wBC and GOLD across four generation tasks.
Problem
RLHF with PPO is unstable and data-hungry, requiring fresh LM-generated data each step; the paper asks whether reward-based learning can be done purely offline on pre-existing language data.
Approach
Treat the full output sequence as a single action and derive an offline policy gradient weighted by the reference policy's advantage (reward minus a learned value estimate from frozen reference LM features) and an importance weight, training only on positive-advantage examples.
Key results
- On HHA, A-LoL seq. reaches average reward 57.6 vs DPO 58.6 (best overall 58.6), with highest diversity (Distinct-3 .727) and GPT-4 safety win% 73.2 vs DPO 60.2.
- On Reddit multi-reward task trained on downvoted data, A-LoL seq. reaches reward 3.18 vs R-GOLD 3.05 and PPO 3.06, with much higher distinct-n.
- On COMET commonsense task, A-LoL seq. raises COMET critic score from 84.6 to 93.0 vs wBC 88.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use advantage-based training with full-sequence importance weights instead of reward-based LoL | LLaMA-7B on HHA · HHA test | Avg. Reward: 55.1_{\pm0.6 } -> 57.3_{\pm0.3 } | Helped |
| Set the importance weight to 1 (ablation) | A-LoL on LLaMA-7B · HHA test | Avg. Reward: 57.3_{\pm0.3 } -> 56.4_{\pm0.6 } | Hurt |
| Replace full-sequence importance weights with per-token importance weights (ablation) | A-LoL on LLaMA-7B · HHA test | Distinct-1/2/3: $.094$/$ .403$/$ .657$ -> $.105$/$\mathbf{.464}$/$\mathbf{.727}$ | Helped |
| Replace importance weighting with a weighted KL penalty (ablation) | A-LoL on LLaMA-7B · HHA validation | reward | Hurt |
| Use per-token importance weights instead of full-sequence importance weights (ablation) | A-LoL on DialoGPT-medium · Reddit upvoted replies test | Reward: $3.15$ -> $\mathbf{3.28}$ | Helped |
| Use advantage-based training rather than reward-based LoL | DialoGPT-medium on Reddit upvoted replies · Reddit upvoted replies test | Reward: $2.95$ -> $3.15$ | Helped |
| Use per-token importance weights instead of full-sequence importance weights (ablation) | A-LoL on DialoGPT-medium · Reddit downvoted replies test | Reward: $3.14$ -> $\mathbf{3.18}$ | Helped |
| Use advantage-based A-LoL training on downvoted data rather than the reference policy | DialoGPT-medium · Reddit downvoted replies test | Reward: $2.87$ -> $3.14$ | Helped |
| Use A-LoL sequence instead of Reward LoL | COMET DIS TIL · Commonsense Reasoning ATOMIC20 test | COMET Critic: $89.4$ -> $\mathbf{93.0}$ | Helped |
| Use advantage-based A-LoL instead of reward-based LoL | DialoGPT-medium on WoW training · Knowledge-grounded dialog FaithDial test | Reward: $2.72$ -> $2.83$ | Helped |
| Use priority sampling instead of random sampling (ablation) | A-LoL · HHA validation | reward | Helped |
| Remove negative-advantage instances instead of sampling the entire dataset (ablation) | A-LoL · HHA validation | reward | Helped |
Surprising findings
- Training on downvoted Reddit data with A-LoL nearly closes the gap to training on upvoted data, because about 48% of downvoted examples get negative advantage and are discarded.
- PPO on Reddit reward hacks the summed reward, producing generic responses with very low distinct-n scores.
Limitations
- DPO and preference-based methods cannot be applied to multi-reward or preference-free tasks, limiting the comparison scope.
- Full-sequence importance weights cause instability; clipping is needed, and without clipping training degenerates.
- Experiments are limited to models up to 7B and rely on learned reward models or classifiers that may be misaligned.
Experimental setup
Datasets: Helpful and Harmless Assistant (HH-RLHF) Reddit comment-response pairs ATOMIC10x Wizard of Wikipedia FaithDial
Benchmarks: HH-RLHF test set Reddit response test set ATOMIC 2020 test set FaithDial test set
Metrics: average reward from reward model Distinct-1/2/3 average length human and GPT-4 win rate COMET critic score coverage density
Baselines: PPO DPO DPO (ref. free) PRO wBC Reward GOLD R-LoL NLL
Builds on: PPO DPO PRO GOLD weighted behavior cloning
Closest papers by meaning
- Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.91)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.91)
- Explainable Reinforcement Learning from Human Feedback to Improve Alignment (NeurIPS 2025 · similarity 0.90)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.90)
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (ICML 2024 · similarity 0.90)
- Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.