Research map / Reinforcement learning / Offline RL
Active Reinforcement Learning Strategies for Offline Policy Improvement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ActiveRL, an uncertainty-driven active trajectory collection method that selects initial states and explores uncertain regions to augment offline RL datasets under a small interaction budget.
Problem
Offline RL agents often have poor state-space coverage, and online interaction is expensive or constrained; the question is how to collect minimal additional environment trajectories that most improve an offline-learned policy.
Approach
Learn an ensemble of contrastive state/state-action representation models and use their disagreement as epistemic uncertainty to (i) choose uncertain candidate start states and (ii) pick uncertain actions during epsilon-greedy exploration, truncating trajectories when uncertainty drops; the policy is then retrained on the augmented data.
Key results
- Across environments, the active method reaches combined total score 1372.9 versus 828.7 for Offline+FT and 781.9 for Offline+RND.
- Reduces additional online interaction by up to 75% (e.g., maze2d-large-easy 75%) relative to the best Offline+FT baseline.
- Ablation on maze2d-large-hard: active initial-state selection with uncertainty exploration reaches 176.3 vs 1.0-ish for fine-tuning variants; ablations show both components individually help.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use active trajectory collection instead of fine-tuning | Offline policy (TD3+BC, IQL, CQL, or BPPO) · Maze2d, AntMaze, locomotion, CARLA, and Unitree Go1 benchmarks | mean normalized score: 134.3 vs 77.5; 197.3 vs 21.7; 201.7 vs 6.0; 88.1 vs 86.7; 71.6 vs 56.0; 73.1 vs 68.9; 73.8 vs 68.5; 22.8 vs 19.9; 22.9 v | Helped |
| Add active initial state selection to random exploration (ablation) | ActiveRL · maze2d-large-easy and maze2d-large-hard | mean normalized score: 45.5 -> 88.1; 25.0 -> 74.6 | Helped |
| Add active initial state selection to offline-policy exploration (ablation) | ActiveRL · maze2d-large-easy and maze2d-large-hard | mean normalized score: 0.7 -> 92.9; 0.2 -> 139.9 | Mixed |
| Add active initial state selection to uncertainty-based exploration (ablation) | ActiveRL · maze2d-large-easy and maze2d-large-hard | mean normalized score | Helped |
| Replace random exploration with uncertainty-based exploration using active initial state selection (ablation) | ActiveRL · maze2d-large-easy and maze2d-large-hard | mean normalized score: 88.1 -> 133.8; 74.6 -> 176.3 | Helped |
| Replace offline-policy exploration with uncertainty-based exploration using active initial state selection (ablation) | ActiveRL · maze2d-large-easy and maze2d-large-hard | mean normalized score: 92.9 -> 133.8; 139.9 -> 176.3 | Helped |
| Use ActiveRL rather than TD3 for online learning from scratch (ablation) | Online TD3+BC-based ActiveRL · maze2d-umaze-v1, maze2d-medium-v1, maze2d-large-v1, and unitree-go1-hard | mean normalized score: 142.79 -> 164.8; 148.1 -> 178.8; 98.7 -> 169; 43.9 -> 52.8 | Helped |
Surprising findings
- Uniform action sampling for exploration fails in high-dimensional locomotion, while the uncertainty-based exploration policy works.
- Ensemble disagreement via representation models outperformed Q-value variance-based uncertainty in the maze2d ablation (max squared difference 176.3 vs var 144.6 on maze2d-large-hard).
Limitations
- Relies on D4RL-style pruned/subsampled datasets and small-scale simulators; results depend on hand-tuned epsilon and beta per environment.
- Requires a fixed set of candidate initial states from the environment and, in restricted settings, an additional goal-based policy and graph clustering.
- Some locomotion comparisons are inconclusive (walker2d-medium reduction reported as '-').
Experimental setup
Datasets: D4RL maze2d (pruned) D4RL antmaze (30% subsampled) D4RL locomotion (subsampled) CARLA expert dataset Unitree Go1 offline dataset
Benchmarks: Maze2d AntMaze Gym-MuJoCo locomotion CARLA IsaacSim Go1
Metrics: D4RL normalized score percentage of fewer environment interactions
Baselines: Behavior Cloning Offline Offline + FT Offline + RND TD3 Random exploration Offline-policy exploration
Builds on: TD3+BC IQL CQL BPPO Random Network Distillation
Closest papers by meaning
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.89)
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
- Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
- Rethinking Optimal Transport in Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- The Generalization Gap in Offline Reinforcement Learning (ICLR 2024 · similarity 0.87)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.