AI Paper Atlas

Research map / Reinforcement learning / Policy gradient and actor-critic

Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning

Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, Sergey Levine

ICLR 2021 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 6 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add the singular-value rank penalty to DQNDQN · Atari 5% offline replay, 16 gamesmedian performance: 74.5% median improvement; 16/16 gamesImproved
Add the singular-value rank penalty to CQLCQL · Atari 5% offline replay, 16 gamesmedian performance: 14.1% median improvement; 11/16 gamesImproved
Add the singular-value rank penaltyRainbow (n=4) · Atari online data-efficient RLmedian performance: 20.66% median improvement; 12/16 gamesImproved
Add the singular-value rank penaltyDQN · Online data-efficient RLmedian performance: 11.5%Hurt
Increase gradient updates per environment step (ablation)DQN, Rainbow, and neural FQI · Online data-efficient RLperformanceHurt
Replace bootstrapped targets with Monte Carlo returns (ablation)Offline policy evaluation · Atarieffective feature rankImproved
Use fitted Q-evaluation bootstrapping instead of Q-learning backups (ablation)Offline policy evaluation · Atarieffective feature rankNo clear effect
Reinitialize the Q-network at each fitting iteration (ablation)Neural FQI · Gridworldeffective feature rankNo clear effect
Regress to oracle Q-values rather than bootstrapped TD estimates (ablation)Gridworld control · Gridworldeffective feature rankImproved
Increase the offline dataset from 5% to 20% (ablation)DQN and FQE · Atari offline RLeffective feature rankNo clear effect
Use one-hot features instead of the usual features (ablation)FQI · GridworldQ* fitting errorImproved
Increase gradient updates per environment step (ablation)SAC · Online HalfCheetah-v2, Hopper-v2, and Walker2d-v2performanceImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy