Research map / Reinforcement learning / Policy gradient and actor-critic
Bigger, Regularized, Optimistic: scaling for compute and sample efficient continuous control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that scaling a regularized critic (BroNet with LayerNorm, weight decay, resets) combined with optimistic exploration and non-pessimistic quantile Q-values yields BRO, a model-free SAC-based agent that matches or beats TD-MPC2 and other baselines on 40 DMC, MetaWorld and MyoSuite tasks.
Problem
Sample-efficient continuous control: naive critic scaling often degrades performance in deep RL, and model-free methods lag model-based approaches on hard tasks such as Dog and Humanoid within 1M environment steps.
Approach
Pair a regularized, scaled critic architecture (BroNet with LayerNorm residual blocks) with weight decay and periodic full-parameter resets, then add optimistic exploration via a dual actor with quantile-based epistemic uncertainty and drop clipped double Q-learning, so that critic scaling substitutes for many domain-specific RL inductive biases.
Key results
- BRO achieves over 90% success in MetaWorld and MyoSuite and over 85% of maximal returns on DMC per the conclusion, with near-optimal Dog and Humanoid policies.
- BRO solves 33 of 40 tasks above 80% of maximal performance versus 18 of 40 for SR-SAC.
- TD-MPC2 needs roughly 2.5x more environment steps to match BRO on Dog and Humanoid tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale the regularized critic network (ablation) | BRO · 40 continuous-control tasks | performance | Helped |
| Increase replay ratio from 1 to the BRO setting (ablation) | BRO · 40 continuous-control tasks | performance | Helped |
| Use risk-neutral Q-values instead of pessimistic Clipped Double Q-learning (ablation) | BRO · 40 continuous-control tasks | performance | Helped |
| Add dual-actor optimistic exploration (ablation) | BRO · 40 continuous-control tasks | performance: around 10% | Helped |
| Use quantile Q-values (ablation) | BRO · Continuous-control tasks, especially smaller networks | performance | Helped |
| Add weight decay (ablation) | BRO · 40 continuous-control tasks | performance | Helped |
| Reduce replay ratio from 2 to 1 (ablation) | BRO (Fast) · 40 continuous-control tasks | performance | No clear effect |
| Reduce batch size from 256 to 128 (ablation) | BRO · 40 continuous-control tasks | performance | No clear effect |
| Add full-parameter resets (ablation) | BRO · Continuous-control tasks | performance | Helped |
| Scale the actor network (ablation) | Off-policy continuous-control actor-critic algorithms · Continuous-control tasks | performance | Hurt |
| Use target networks (ablation) | BRO and BRO (Fast) · DMC, MyoSuite, and other tested environments | performance | Helped |
| Replace BroNet with Spectral architecture (ablation) | BRO · DeepMind Control Suite Dog environments | final return: 763.5 -> 73.5 | Hurt |
Surprising findings
- Using the ensemble mean instead of the clipped minimum (disabling clipped double Q-learning) improves performance under network regularization without increasing overestimation.
- Algorithmic enhancements matter less as critic size grows; the largest models gain little from smaller batches, quantile Q-values, or optimism.
Limitations
- Higher memory requirements and possible inference latency for high-frequency control due to larger model size.
- Not evaluated in discrete action settings.
- Evaluated only on proprioceptive state inputs, not image-based RL.
Experimental setup
Datasets: DeepMind Control Suite MetaWorld MyoSuite
Benchmarks: DeepMind Control Suite MetaWorld MyoSuite AntMaze Adroit
Metrics: interquartile mean return success rate
Baselines: TD-MPC2 SR-SAC CrossQ SAC TD3
Builds on: Soft Actor-Critic CrossQ SR-SAC TD-MPC2 Clipped Double Q-learning
Closest papers by meaning
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data (ICML 2024 · similarity 0.88)
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.88)
- FASTER: Value-Guided Sampling for Fast RL (NeurIPS 2026 · similarity 0.88)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.87)
- Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.