Research map / Reinforcement learning / Policy gradient and actor-critic
Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the BEE operator, blending an in-sample Bellman exploitation target (expectile-based max over replay buffer actions) with a SAC-style exploration target, yielding BAC (model-free) and MB-BAC (model-based) that beat baselines on 50+ continuous control tasks and real quadruped tasks.
Problem
In the later stage of off-policy actor-critic training, Q-values are underestimated because Bellman targets use actions from the current suboptimal policy rather than better actions stored in the replay buffer.
Approach
A Blended Exploitation and Exploration (BEE) Bellman operator linearly combines an exploitation target that maximizes Q over replay-buffer actions (via expectile regression on a state value function) with an exploration target from the current policy with SAC entropy, integrated into SAC/TD3 and MBPO.
Key results
- BAC reaches average return up to 280,000 at 2.5M steps and 360,000 at 5M steps on HumanoidStandup, 1.5x and 2.1x the strongest baseline.
- On DKittyWalk tasks BAC achieves a success rate roughly 50% higher than SAC.
- BAC is claimed to outperform SOTA methods on over 50 continuous control tasks with a single hyperparameter set (lambda around 0.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the BEE operator instead of the standard actor-critic update | BAC vs. the strongest baseline · HumanoidStandup, 2.5M steps | average return: 1.5x higher | Helped |
| Use the BEE operator instead of the standard actor-critic update | BAC vs. the strongest baseline · HumanoidStandup, 5M steps | average return: 2.1x higher | Helped |
| Use BAC instead of SAC (ablation) | DKittyWalk-Medium · DKittyWalk-Medium | recovery speed after adding expert trajectories | Helped |
| Use the BEE operator in TD3 (BEE-TD3) | TD3 · 15 DMControl tasks | performance | Helped |
| Use the BEE operator in TD3 (BEE-TD3) (ablation) | TD3 · D'Kitty tasks | success rate and average return | Helped |
| Replace entropy regularization with target policy smoothing regularization (ablation) | BEE (TD3-BEE) · Walker2d-v2 | performance | Helped |
| Set the exploitation-exploration weight λ around 0.5 (ablation) | BAC · 35 locomotion and manipulation tasks | performance | Helped |
| Set λ to 0, reducing BAC to the online version of IQL (ablation) | BAC · continuous-control benchmarks | performance | Mixed |
Surprising findings
- Q-value underestimation, not overestimation, dominates in the later training stage of SAC and TD3 even when the double-Q trick is removed.
Limitations
- Exploitation relies on in-sample max over replay buffer, which may be less effective when buffer lacks good actions.
- Real-world evaluation is limited to a single quadruped robot (D'Kitty) and a few terrains.
Experimental setup
Datasets: MuJoCo DMControl Meta-World ManiSkill2 Adroit MyoSuite ROBEL Shadow Dexterous Hand DKittyWalkRandomDynamics
Benchmarks: MuJoCo DMControl Meta-World ManiSkill2 Adroit MyoSuite ROBEL Shadow Dexterous Hand DKittyWalk-Medium DKittyWalk-Hard
Metrics: average return success rate normalized Q estimation vs Monte-Carlo value
Baselines: SAC TD3 DAC RRS PPO MBPO CMLO AutoMBPO SLBO PETS
Builds on: SAC TD3 MBPO IQL Diversity Actor-Critic
Closest papers by meaning
- Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · similarity 0.89)
- Efficient Off-Policy Learning for High-Dimensional Action Spaces (ICLR 2025 · similarity 0.89)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Horizon Adaptive Offline Policy Learning via Value Stitching (NeurIPS 2026 · similarity 0.88)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.88)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.