Research map / Reinforcement learning / Policy gradient and actor-critic
Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, Sergey Levine
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the singular-value rank penalty to DQN | DQN · Atari 5% offline replay, 16 games | median performance: 74.5% median improvement; 16/16 games | Improved |
| Add the singular-value rank penalty to CQL | CQL · Atari 5% offline replay, 16 games | median performance: 14.1% median improvement; 11/16 games | Improved |
| Add the singular-value rank penalty | Rainbow (n=4) · Atari online data-efficient RL | median performance: 20.66% median improvement; 12/16 games | Improved |
| Add the singular-value rank penalty | DQN · Online data-efficient RL | median performance: 11.5% | Hurt |
| Increase gradient updates per environment step (ablation) | DQN, Rainbow, and neural FQI · Online data-efficient RL | performance | Hurt |
| Replace bootstrapped targets with Monte Carlo returns (ablation) | Offline policy evaluation · Atari | effective feature rank | Improved |
| Use fitted Q-evaluation bootstrapping instead of Q-learning backups (ablation) | Offline policy evaluation · Atari | effective feature rank | No clear effect |
| Reinitialize the Q-network at each fitting iteration (ablation) | Neural FQI · Gridworld | effective feature rank | No clear effect |
| Regress to oracle Q-values rather than bootstrapped TD estimates (ablation) | Gridworld control · Gridworld | effective feature rank | Improved |
| Increase the offline dataset from 5% to 20% (ablation) | DQN and FQE · Atari offline RL | effective feature rank | No clear effect |
| Use one-hot features instead of the usual features (ablation) | FQI · Gridworld | Q* fitting error | Improved |
| Increase gradient updates per environment step (ablation) | SAC · Online HalfCheetah-v2, Hopper-v2, and Walker2d-v2 | performance | Improved |
Closest papers by meaning
- Resetting the Optimizer in Deep RL: An Empirical Study (NeurIPS 2023 · similarity 0.89)
- Understanding and Leveraging Overparameterization in Recursive Value Estimation (ICLR 2022 · similarity 0.88)
- DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization (ICLR 2022 · similarity 0.88)
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (ICML 2024 · similarity 0.88)
- Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn (NeurIPS 2024 · similarity 0.87)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
