Research map / Reinforcement learning / Policy gradient and actor-critic
Compute-Optimal Scaling for Value-Based Deep RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies compute-optimal allocation between critic width and UTD ratio (plus batch size) for value-based off-policy RL, identifying TD-overfitting and fitting power-law scaling laws on DMC and HumanoidBench.
Problem
How to allocate a fixed compute or compute-plus-data budget between Q-network model size and the updates-to-data (UTD) ratio in value-based deep RL, given that batch size interacts with both.
Approach
Identifies TD-overfitting, where small critics produce poor TD-targets so large batch sizes hurt validation TD-error, and uses this to model a maximal admissible batch size as a function of UTD and model width; then fits power-law scaling laws for data, compute and budget-optimal UTD and width.
Key results
- Batch-size rule achieves 48.9% average relative error versus 55.1% for a log-linear fit.
- Data-efficiency fit achieves 10.0% relative error on fitted UTD/width configurations, 14.9% on interpolation, 18.0% on extrapolation.
- Compute-optimal (UTD, width) scaling needs 1.26x data relative to sigma-only scaling and 1.11x relative to N-only scaling (average).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed compute-optimal UTD and model-size prescriptions instead of scaling UTD only | BRO · 4-task average | data efficiency ratio: 1.26 -> 1.00 | Helped |
| Use the proposed compute-optimal UTD and model-size prescriptions instead of scaling model size only | BRO · 4-task average | data efficiency ratio: 1.11 -> 1.00 | Helped |
| Add a batch size selected for the compute budget instead of using a constant fixed batch size (ablation) | Compute-optimal UTD and model-size scaling · 4-task average | data efficiency ratio: 1.03 -> 1.00 | Helped |
| Use the proposed batch size rule instead of a log-linear batch size fit | BRO · 4-task average | relative error: 55.1% -> 48.9% | Helped |
| Increase model width | BRO Q-network · h1-crawl | validation TD-error | Helped |
| Increase batch size with widths 256 or 512 | BRO Q-network · DMC and HumanoidBench tasks | validation TD-error | Mixed |
| Increase batch size with larger models | BRO Q-network · DMC and HumanoidBench tasks | validation TD-error | No clear effect |
| Generate TD-targets with a larger main Q-network rather than a low-capacity one (ablation) | Passive critic diagnostic · h1-crawl | validation TD-error | Helped |
Surprising findings
- Larger batch sizes increase validation TD-error for small critics, even though they reduce training TD-error (TD-overfitting).
- Validation TD-error does not show classical overfitting for larger models; larger models lower both training and validation TD-error.
Limitations
- Limited to simulated robotic control tasks; no visual or language domains.
- Only batch size, UTD and width studied; learning rate and critic update ratio left for future work.
- Relative error of batch size fit is high (~49%), reflecting a wide range of near-optimal batch sizes.
Experimental setup
Datasets: DeepMind Control Suite HumanoidBench
Benchmarks: DeepMind Control Suite HumanoidBench
Metrics: validation TD-error data efficiency ratio relative error of scaling fit Monte Carlo return
Baselines: UTD-only scaling model-only scaling fixed batch size compute-optimal log-linear batch size fit
Builds on: BRO SimbaV2 SAC Scaling laws for UTD in off-policy RL
Closest papers by meaning
- Value-Based Deep RL Scales Predictably (ICML 2025 · similarity 0.90)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.90)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.89)
- Bridging Compute- and Data-Optimal Pretraining (NeurIPS 2026 · similarity 0.88)
- Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models (ICLR 2025 · similarity 0.88)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.