Research map / Reinforcement learning / Offline RL
Offline Actor-Critic Reinforcement Learning Scales to Large Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows offline actor-critic RL (a KL-regularized, BC-anchored variant of MPO/DIME) on a Perceiver-based transformer scales to ~1B parameters, follows Chinchilla-style scaling laws, and outperforms BC baselines on 132 multi-task control and robotics tasks.
Problem
Scaling offline reinforcement learning to large transformer policies trained on heterogeneous multi-task control data (sub-optimal and expert), where behavioral cloning is the dominant paradigm.
Approach
An offline actor-critic with a KL-regularized policy objective converted into weighted supervised learning, mixed with a BC term (alpha) to stabilize training, combined with a Perceiver-IO style architecture (latent cross-attention, multi-scale proprioception normalizer, action-conditioned Q-query cross-attention) so that large models remain efficient.
Key results
- alpha-PAC reaches 92.1% of expert performance on 32 Control Suite tasks vs 63.6% for the BC Gato baseline, and 94.3% after RL finetuning.
- On real CHEF:real stacking, alpha-PAC reaches 69.8% success vs 7.1% for BC+Q, rising to 93.2% after three rounds of RL finetuning with self-generated data.
- Scaling fits show offline RL (PAC) has a higher parameter-scaling exponent (a=0.920) than BC+Q (a=0.828), suggesting RL scales better with compute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use offline actor-critic instead of BC+Q | PAC · Gato:Control | expert average reward (%): 84.6 -> 87.7 | Helped |
| use dataset-specific α | PAC · Gato:Control | expert average reward (%): 87.7 -> 92.1 | Helped |
| use offline actor-critic instead of BC+Q | PAC · RC:Tower | success rate (%): 71.3 -> 69.3 | Hurt |
| use dataset-specific α | PAC · RC:Insertion | success rate (%): 80.3 -> 89.3 | Helped |
| use offline actor-critic instead of BC+Q | PAC · CHEF:sim | success rate (%): 11.0 -> 55.0 | Helped |
| use offline actor-critic instead of BC+Q | α-PAC · CHEF:real | success rate (%): 7.1 -> 69.8 | Helped |
| fine-tune with self-generated data | α-PAC · CHEF:real | success rate (%): 69.8 -> 93.2 | Helped |
| fine-tune PAC with 10,000 episodes per task | α-PAC · Gato:Control | expert average reward (%): 92.1% -> 94.3% [91.3, 97.3] | Helped |
| use offline RL rather than BC at matched compute | PAC · multi-task control (average return) | average return | Helped |
| switch from α = 1 to α = 0 during training | BC+Q then PAC · CHEF:real | success rate (%): 61.9% [60.0, 63.8] | Helped |
Surprising findings
- Training loss was not a reliable indicator of model performance in this setting, so return profiles were used for scaling-law fits.
- Using a Q-value loss term in BC never decreased performance, so BC+Q was kept as a baseline.
Limitations
- Offline RL requires reward annotations, which can be costly.
- Little evidence of transfer across the wide variety of domains considered.
Experimental setup
Datasets: Gato Control Suite data RoboCat RGB Stacking and Insertion data CHEF sim and real data
Benchmarks: Control Suite RGB Stacking (RoboCat) CHEF
Metrics: success rate percentage of expert return
Baselines: BC+Q FilteredBC Gato RoboCat CQL Q-Transformer
Builds on: MPO DIME Perceiver-IO Gato Chinchilla scaling laws
Closest papers by meaning
- Actor-Critic Alignment for Offline-to-Online Reinforcement Learning (ICML 2023 · similarity 0.92)
- Online Meta-Critic Learning for Off-Policy Actor-Critic Methods (NeurIPS 2020 · similarity 0.91)
- Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning (NeurIPS 2021 · similarity 0.90)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.90)
- GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation (AAAI 2024 · similarity 0.90)
- Scaling Offline Evaluation of Reinforcement Learning Agents through Abstraction (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.