Research map / Reinforcement learning / Offline RL
Is Value Learning Really the Main Bottleneck in Offline RL?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically studies whether value learning, policy extraction, or test-time policy generalization bottlenecks offline RL, using data-scaling matrices across eight environments, and proposes two test-time policy improvement methods (OPEX and TTT).
Problem
Offline RL often underperforms imitation learning, and it is unclear whether the bottleneck is value learning, policy extraction from the value function, or policy generalization to test-time states.
Approach
Uses data-scaling matrices that vary value-data and policy-data amounts separately to show policy extraction (especially behavior-regularized policy gradient like DDPG+BC over AWR) often matters more than value learning, and that evaluation-state (test-time) MSE is the main bottleneck; proposes on-the-fly policy extraction (OPEX) and test-time training (TTT) to address it.
Key results
- DDPG+BC matches or beats AWR in 15 of 16 settings in the data-scaling aggregate table (e.g., exorl-walker IQL 191 vs AWR 99).
- In offline-to-online experiments, evaluation MSE improves while training and validation MSE stay flat, and evaluation MSE is the most predictive of performance.
- OPEX and TTT improve over vanilla IQL and SfBC on many tasks, often by significant margins.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace AWR with DDPG+BC for policy extraction | Offline RL with IQL or SARSA · exorl-walker | return: 99 ± 1 -> 191 ± 6 | Helped |
| Replace AWR with DDPG+BC for policy extraction | Offline RL with IQL or SARSA · exorl-cheetah | return: 71 ± 1 -> 101 ± 2 | Helped |
| Replace AWR with DDPG+BC for policy extraction | Offline RL with IQL · gc-antmaze-large | return: 51 ± 2 -> 58 ± 2 | Helped |
| Replace AWR with DDPG+BC for policy extraction | Offline RL with IQL · d4rl-hopper | return: 53 ± 1 -> 52 ± 3 | No clear effect |
| Replace AWR with DDPG+BC for policy extraction | Offline RL with IQL · gc-roboverse | return: 23 ± 2 -> 20 ± 2 | Hurt |
| Use DDPG+BC rather than AWR for policy extraction | Offline RL across the evaluated settings · The 16 task/value-algorithm settings in the data-scaling analysis | return | Helped |
| Increase data coverage, allowing more suboptimal exploratory data | Offline RL agents trained on IQL · gc-antmaze-large and adroit-pen | return | Helped |
| Apply OPEX or TTT during evaluation | Offline IQL policy · Eight evaluated tasks | return | Helped |
Surprising findings
- Policy extraction choice often affects performance more than the value learning objective, despite the policy being subordinate to the value function.
- Offline RL policies saturate on in-distribution states; performance gains come mainly from improving accuracy on test-time states.
Limitations
- MSE metrics against expert policies are proxies for policy accuracy and may not reflect performance when multiple optimal actions exist or experts are suboptimal.
- Analysis restricted to continuous-action environments; DDPG+BC not directly applicable to discrete actions.
- Test-time generalization is theoretically hard to guarantee without coverage or structural assumptions.
Experimental setup
Datasets: D4RL antmaze-large-diverse-v2 D4RL hopper-medium-v2 D4RL walker2d-medium-v2 D4RL kitchen D4RL adroit pen/hammer/door ExORL walker-run ExORL cheetah-run roboverse goal-conditioned image dataset
Benchmarks: D4RL ExORL gc-antmaze-large gc-roboverse
Metrics: normalized return / success rate training MSE validation MSE evaluation MSE
Baselines: AWR DDPG+BC SfBC IQL SARSA CRL
Builds on: IQL AWR DDPG+BC SfBC Contrastive RL
Closest papers by meaning
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.92)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- The Generalization Gap in Offline Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.91)
- Scaling Offline Evaluation of Reinforcement Learning Agents through Abstraction (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.