Research map / Reinforcement learning / Offline RL
Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OBAC, a model-free off-policy actor-critic that trains an implicit offline-optimal policy (IQL-style expectile regression) on the shared replay buffer and uses it as an adaptive behavior-cloning constraint when its state value exceeds the online policy's.
Problem
Off-policy RL under-exploits the replay buffer, limiting sample efficiency; it is unclear when and how an offline policy learned from the online buffer can improve the online learner.
Approach
Concurrently learn an offline-optimal policy from the shared replay buffer without explicitly representing it (via expectile-based value learning), compare state values of online and offline policies, and apply a behavior-cloning constraint toward buffer actions only in states where the offline value is higher.
Key results
- OBAC is evaluated on 53 continuous control tasks across 6 suites, outperforming model-free baselines SAC, TD3 and BAC.
- OBAC is reported to be comparable to TD-MPC2 with about 50% of its parameters and about 20% of its training wall time per 1M steps.
- A fixed (always-on) constraint degrades performance relative to the adaptive constraint in ablations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OBAC instead of BAC | Noisy Hammer and Pen tasks · Hammer and Pen (σ=0.1) | decline rate: Hammer: 16.67% -> 11.11%; Pen: 66.67% -> 25.00% | Helped |
| Use OBAC instead of SAC | Noisy Hammer and Pen tasks · Hammer and Pen (σ=0.1) | decline rate: Hammer: 12.5% -> 11.11%; Pen: 70.00% -> 25.00% | Helped |
| Add a value-based adaptive policy constraint (ablation) | OBAC · Mujoco and DMControl ablation tasks | policy performance | Helped |
| Replace the adaptive constraint with a fixed replay-buffer constraint (ablation) | OBAC · Mujoco and DMControl ablation tasks | policy performance | Hurt |
| Remove the policy constraint (ablation) | OBAC · Mujoco and DMControl ablation tasks | policy performance | Mixed |
| Use a deterministic online policy instead of a stochastic Gaussian policy (ablation) | OBAC · Various scenarios | policy performance | Helped |
Surprising findings
- A concurrently trained offline policy on the online replay buffer can sometimes outperform the online SAC policy, even though it never interacts with the environment.
- TD-MPC2 performs poorly on Mujoco tasks, which the authors attribute to early termination (done-signal) settings.
Limitations
- Constraint and proposition rely on heuristic approximations (e.g., expectile tau->1 approximation of max over buffer actions, Gaussian policy projection), so theoretical guarantees are only loosely tied to the practical algorithm.
- Gains depend on whether the offline policy is superior, which varies across tasks and training stages.
- Behavior clone weight lambda is fixed rather than auto-tuned.
Experimental setup
Datasets: online replay buffer collected during training Mujoco DMControl Meta-World Adroit Maniskill2 Myosuite
Benchmarks: Mujoco DMControl Meta-World Adroit Maniskill2 Myosuite
Metrics: average return success rate sample efficiency wall-clock training time parameter count
Baselines: SAC TD3 BAC TD-MPC2
Builds on: Soft Actor-Critic Implicit Q-Learning BAC TD3 Clipped Double Q-learning
Closest papers by meaning
- Offline RL Without Off-Policy Evaluation (NeurIPS 2021 · similarity 0.92)
- Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization (ICLR 2024 · similarity 0.91)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.91)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees (ICLR 2024 · similarity 0.91)
- Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.