Research map / Reinforcement learning / Policy gradient and actor-critic
Value-Based Deep RL Scales Predictably
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that value-based off-policy deep RL (SAC, BRO, PQL) scales predictably: data and compute requirements lie on a UTD-controlled Pareto frontier, and batch size and learning rate follow power laws in UTD, enabling extrapolation to larger data, compute, and budgets.
Problem
Value-based off-policy deep RL is commonly considered unpredictable at scale; it is unclear how data, compute, and hyperparameters (batch size, learning rate, UTD ratio) should be set so that large-scale performance can be predicted from small runs.
Approach
Model data and compute requirements as power-law functions of the updates-to-data (UTD) ratio, with batch size and learning rate predicted as power laws in UTD, so that the UTD ratio traces a data-compute Pareto frontier and yields predictable budget-optimal hyperparameters.
Key results
- Pareto frontier extrapolation error is 7.8% toward larger compute and 10.6% toward larger data.
- Best batch size and learning rate follow power laws in UTD, with near-zero correlation between best batch size and learning rate on DMC (R=0.04).
- Optimal UTD follows a power law in budget (sigma* ~ 1.4e8 * F^-0.53 on DMC), extrapolated to held-out budgets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UTD-dependent batch size and learning rate instead of hyperparameters tuned at σ=2 for all UTD values | SAC · OpenAI Gym | data efficiency | Helped |
| Use the proposed data-efficiency power-law fit to extrapolate toward larger compute | Proposed scaling fit · OpenAI Gym | Pareto frontier extrapolation error: 7.8% | Helped |
| Use the proposed data-efficiency power-law fit to extrapolate toward larger data | Proposed scaling fit · OpenAI Gym | Pareto frontier extrapolation error: 10.6% | Helped |
| Add an additive offset to the batch-size and learning-rate power-law fits (ablation) | Proposed hyperparameter fit · Value-based deep RL | extrapolation accuracy | Hurt |
Surprising findings
- Critical batch size from policy-gradient literature has little correlation with empirically optimal batch size in value-based RL.
- Hyperparameters tuned at a single UTD value (sigma=2) perform worse at other UTD values, with the gap growing for larger UTD.
Limitations
- Only three hyperparameters studied; model size and weight decay/reset frequency not optimized.
- Relatively small models and limited compute range (not spanning many orders of magnitude).
- Power-law functional forms lack theoretical grounding.
Experimental setup
Datasets: DeepMind Control Suite OpenAI Gym IsaacGym
Benchmarks: DeepMind Control Suite OpenAI Gym IsaacGym Franka-Push
Metrics: data efficiency (data to reach target return) compute (FLOPs) extrapolation error (%) normalized return
Baselines: fixed hyperparameters tuned at UTD=2 BRO PQL
Builds on: Soft Actor-Critic BRO PQL Chinchilla-style scaling laws
Closest papers by meaning
- Compute-Optimal Scaling for Value-Based Deep RL (NeurIPS 2025 · similarity 0.90)
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (ICML 2024 · similarity 0.87)
- Horizon Reduction Makes RL Scalable (NeurIPS 2025 · similarity 0.87)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.87)
- Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models (ICLR 2025 · similarity 0.86)
- The Effective Horizon Explains Deep RL Performance in Stochastic Environments (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.