Research map / Reinforcement learning / RL from feedback
Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes an off-policy REINFORCE variant (AsymRE) whose baseline V controls positive vs. negative emphasis, proving convergence and a phase transition at V = expected behavior reward, and validates on bandits and LLM math reasoning.
Problem
Off-policy RL for LLMs without importance sampling correction is under-studied; the choice of baseline in REINFORCE affects both training dynamics and the limit policy when data comes from an outdated behavior policy.
Approach
Study REINFORCE with an arbitrary baseline V applied off-policy (no importance sampling) and show that a baseline below the behavior policy's expected reward yields stable, improving limit policies with wide support, while a baseline at or above it causes support collapse; recommend a small conservative offset (about -0.1) in the advantage.
Key results
- Theory: for baseline V below the behavior policy's expected reward V^mu, the limit policy improves on mu and keeps wide support; for V >= V^mu the support collapses to a singleton.
- In LLM experiments with delta V = 0, all 7 training runs collapsed, while delta V = -0.1 gave stable training.
- AsymRE compares favorably to GRPO in training speed and final accuracy in the off-policy setting (Llama 8B, Qwen 3B on MATH).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase baseline V while keeping it below the behavior policy's expected reward | AsymRE · 100-arm bandit | expected reward | Helped |
| Set V at or above the behavior policy's expected reward | AsymRE · 100-arm bandit | policy support diversity | Hurt |
| Set V at or above the behavior policy's expected reward | AsymRE policy improvement scheme · 100-arm bandit | expected reward | Hurt |
| Increase δV while keeping it below 0 | AsymRE · MATH | training accuracy | Helped |
| Increase δV while keeping it below 0 | AsymRE · MATH | test accuracy | Helped |
| Set δV to 0 or higher | AsymRE · MATH | training and test loss | Hurt |
| Use δV = -0.1 instead of δV = 0 (ablation) | AsymRE · MATH, Llama 8B | training stability: 7 independent training runs with δV=0; 7 training runs with δV=-0.1 | Helped |
| Use δV = 0 (ablation) | AsymRE · MATH, Llama 8B | training stability: 7 | Hurt |
| Use a baseline slightly below 0 instead of δV = 0 (ablation) | AsymRE · MATH, Llama 8B | training stability: -0.1 | Helped |
| Use AsymRE instead of GRPO | Off-policy LLM training · MATH | training speed and final accuracy | Helped |
| Use V < V^μ rather than V ≥ V^μ | AsymRE policy improvement scheme · 100-arm bandit | expected reward | Helped |
| Use a baseline close to V^μ while remaining strictly below it | AsymRE · 100-arm bandit | expected reward | Helped |
Surprising findings
- Raising the baseline to or above the behavior policy's expected reward causes a sudden loss of policy diversity and training collapse, even though it can look better on the bandit expected reward in the short run.
- Bandit performance is capped at about 0.89 for all baselines, below the optimal arm reward of 0.999.
Limitations
- Analysis is for tabular softmax policies and expected (full-gradient) dynamics; LLM validation is limited and the authors say further large-scale experiments are needed.
- No importance sampling or KL regularization is included, so the method is not asymptotically optimal in general.
- Comparison to GRPO is described as needing further testing for definitive conclusions.
Experimental setup
Datasets: MATH NuminaMath
Benchmarks: MATH NuminaMath
Metrics: accuracy train loss test loss expected reward entropy
Baselines: GRPO
Builds on: REINFORCE GRPO
Closest papers by meaning
- Tapered Off-Policy REINFORCE - Stable and efficient reinforcement learning for large language models (NeurIPS 2025 · similarity 0.92)
- Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models (ICLR 2024 · similarity 0.91)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.90)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.90)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.