Research map / Reinforcement learning / Policy gradient and actor-critic
Adaptive Regularization of Representation Rank as an Implicit Constraint of Bellman Equation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives an upper bound on cosine similarity between consecutive state-action representations from the Bellman equation and uses it as a ReLU penalty (BEER) to adaptively regularize representation rank in DQN and DPG-based deep RL.
Problem
Representation rank in deep RL value networks: unbounded maximization (e.g., InFeR, DR3) can overcomplicate models, so the rank needs adaptive control rather than maximization.
Approach
Derive from the Bellman equation an upper bound on the cosine similarity between representations of consecutive state-action pairs, and use it as a one-sided penalty that only reduces similarity when the bound is violated, thereby adaptively controlling representation rank without unboundedly maximizing it.
Key results
- On 12 challenging DMControl tasks, BEER averages 535.8 peak score versus 335.0 for SAC, 272.2 for InFeR, 265.4 for TD3 and 223.3 for DR3, and wins all 12 tasks.
- On Lunar Lander, BEER shows lower representation rank and lower approximation error than DQN and InFeR in the later training stage.
- On easier DMControl tasks, BEER has only a slight advantage (average 924.4 vs 879.3-915.2 for baselines).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add adaptive Bellman-equation-based representation-rank regularization to DPG | DPG · 12 challenging DMControl tasks (Average) | peak score: 223.3 -> 535.8 | Helped |
| Add adaptive Bellman-equation-based representation-rank regularization to DPG | DPG · 9 easy DMControl tasks (Average) | peak score: 915.2 -> 924.4 | Helped |
| Increase BEER coefficient β from 0.001 to 0.01 (ablation) | BEER · Acrobot Swingup | peak score: 243.3 -> 171.8 | Hurt |
| Increase BEER coefficient β from 0.0005 to 0.01 (ablation) | BEER · Hopper Hop | peak score: 404.5 -> 362.7 | Hurt |
| Increase BEER coefficient β from 0.001 to 0.1 (ablation) | BEER · Pendulum Swingup | peak score: 830.0 -> 616.6 | Hurt |
| Increase BEER coefficient β from 0.01 to 1.0 (ablation) | BEER · Finger Spin | peak score: 981.7 -> 384.1 | Hurt |
| Use β values at or below 0.001 rather than larger regularization coefficients (ablation) | BEER · Four representative DMControl tasks | peak score | Mixed |
| Use plain DPG without TD3 techniques such as double Q clipping and target noise smoothing (ablation) | BEER · DMControl continuous control tasks | peak score | Mixed |
Surprising findings
- Standard value networks (DQN, DR3, InFeR, etc.) often fail to satisfy the derived Bellman-based bound during training, suggesting feature co-adaptation violates it implicitly.
- Pure DPG with BEER outperforms TD3 without TD3's engineering tricks, which the authors use as an ablation.
Limitations
- Theoretical bound relies on assumptions (non-trivial representation and weights) and a semi-gradient approximation of the next-state representation expectation.
- Only evaluated on DMControl and small discrete tasks; authors note plans to validate on other benchmarks.
- Hyperparameter beta selected from a limited sweep on four tasks; results sensitive to beta at higher values.
Experimental setup
Datasets: DeepMind Control Suite Lunar Lander-v1 (gym) Grid World (custom)
Benchmarks: DeepMind Control Suite (12 challenging tasks) DMControl easy tasks (9) Lunar Lander-v1 Grid World
Metrics: peak average return over 1M steps representation rank normalized approximation error
Baselines: DR3 InFeR SAC TD3 DQN
Builds on: DPG TD3 DQN DR3 InFeR
Closest papers by meaning
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.87)
- Decoupling regularization from the action space (ICLR 2024 · similarity 0.85)
- Bigger, Regularized, Optimistic: scaling for compute and sample efficient continuous control (NeurIPS 2024 · similarity 0.85)
- Parseval Regularization for Continual Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning (ICML 2025 · similarity 0.85)
- Shift Before You Learn: Enabling Low-Rank Representations in Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.