Research map / Reinforcement learning / Offline RL
Actor-Critic Alignment for Offline-to-Online Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Actor-critic alignment: reset critic to log π_θ0 + Z_ψ and fine-tune residual R_ϕ online (SAC→ACA) vs direct SAC transfer | SAC+ML offline policy · D4RL locomotion (15 datasets) | total normalized score at 100k online steps: 1249.65 vs 901.54 for SAC→SAC | Helped |
| Use SAC→ACA instead of CQL→BR (balanced replay) | O2O fine-tuning · D4RL locomotion (15 datasets) | total score increase δ: 278.31 vs 190.94 | Helped |
| Alignment step with ODT-style comparison on medium and medium-replay tasks | SAC+ML offline policy · D4RL medium and medium-replay locomotion (200k online) | total score increase δ: 160.27 vs 45.25 for ODT and 4.48 for IQL | Helped |
| Replace offline SAC+ML initialization with CQL initialization for ACA (alignment clips negative log π) (ablation) | ACA · Hopper-medium-v2 | online performance | No clear effect |
| Initialize ACA from CQL instead of SAC+ML while BR is initialized from SAC+ML (ablation) | BR baseline · Hopper-medium-v2 | online performance | Hurt |
| Remove offline data replay during online fine-tuning for ACA (ablation) | SAC→ACA · D4RL locomotion (medium-replay, medium-expert, expert) | total normalized score at 100k online steps: 841.84 without offline data vs 844.98 with | No clear effect |
| Remove Q reparameterization (regular SAC critic with BC actor term) in online ACA (ablation) | SAC→ACA · Hopper-medium-v2 online fine-tuning | online score | Hurt |
| Initialize residual R_ϕ with offline Q instead of baseline Z_ψ (ablation) | SAC→ACA · Hopper-medium-v2 and Walker2d-medium-v2 online fine-tuning | online score | Hurt |
| Use delayed parameters θ⁻ in place of frozen θ0 in log-policy term of Q parameterization (ablation) | SAC→ACA · Hopper-medium-v2 | online score | Hurt |
| Add burn-in period (critic-only online updates for 50k episodes) before fine-tuning (ablation) | SAC→SAC and CQL→SAC · D4RL locomotion medium and expert | initial online performance drop | No clear effect |
| Use SAC+ML offline training instead of TD3+BC | offline policy · D4RL locomotion (15 datasets, total) | total normalized score (offline): 971.3 vs 1013.2 for TD3+BC | No clear effect |
Closest papers by meaning
- Offline Actor-Critic Reinforcement Learning Scales to Large Models (ICML 2024 · similarity 0.92)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.91)
- Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL (NeurIPS 2024 · similarity 0.90)
- Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization (ICLR 2024 · similarity 0.90)
- Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.