Research map / Reinforcement learning / Policy gradient and actor-critic
Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that high-capacity (up to 1B parameter) value models trained online with cross-entropy categorical TD learning, a BroNet residual critic, and learned task embeddings enable effective multi-task RL that beats single-task specialists and transfers sample-efficiently.
Problem
Multi-task online value-based RL is brittle due to sparse rewards, reward-scale imbalance and gradient conflicts, so high-capacity value models have rarely been trained online across many tasks.
Approach
Combine a scaled regularized residual (BroNet) critic with categorical (cross-entropy) TD learning using per-task return normalization, and condition on task embeddings learned by backpropagating the TD loss, instead of separate heads; applied to SAC and DrQ-epsilon.
Key results
- Multi-task BRC matches or surpasses single-task specialists across continuous-control benchmarks while using 40x fewer gradient updates to match the best single-task model on HumanoidBench-Hard.
- Single-task performance saturates around 64M parameters, while the multi-task BRC model keeps improving beyond 250M parameters.
- Using only 300 transitions for embedding-tuning transfer lets BRC exceed 90% success on new ShadowHand shapes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-task BRC instead of single-task specialists | BRC · HumanoidBench-Hard | gradient updates to match best single-task performance: 40× fewer gradient updates | Helped |
| tune only a learned task embedding using 300 transitions while freezing the pretrained network | BRC pretrained on ShadowHand · ShadowHand transfer (29 out-of-distribution objects) | success rate: 300 transitions; over 90% success rates | Helped |
| train online rather than offline | BRC on MW+DMC · MW+DMC | normalized performance: nearly 20% better | Helped |
| increase critic capacity (ablation) | BRC · HumanoidBench-Medium | final performance | Helped |
| use categorical cross-entropy with reward normalization instead of non-distributional MSE (ablation) | BRC · HumanoidBench-Medium | final performance | Helped |
| use learned task embeddings instead of separate heads (ablation) | BRC · HumanoidBench-Medium | final performance | Helped |
| use BroNet rather than SimBa | BRC critic · proprioception-based control | performance | Helped |
| increase model capacity beyond 64M parameters | BRC multi-task · multi-task RL | performance: beyond 250M parameters | Helped |
| increase the number of pretraining tasks | BRC · HumanoidBench-Medium | performance on pretraining tasks | Helped |
| initialize from multi-task pretrained weights rather than training from scratch | BRC single-task learner · transfer tasks | sample efficiency | Helped |
| initialize from multi-task pretrained weights rather than a replay buffer of pretraining transitions | BRC single-task learner · transfer tasks | returns during training | Helped |
| scale the model in mixed-embodiment task suites without task synergy | BRC · mixed-embodiment multi-task RL | learning rate | Hurt |
Surprising findings
- Gradient conflict rate decreases as parameter count increases, suggesting gradient surgery may be unnecessary at large scale.
- Model transfer (initializing from pretrained weights) is more practical than data transfer from a pretraining replay buffer.
Limitations
- Unclear when tasks help or hinder each other, especially for mixed-embodiment suites where synergy is absent in some cases.
- Offline BRC slightly underperforms TD-MPC2 on the MW+DMC offline setting, and offline multi-task RL remains challenging.
Experimental setup
Datasets: DeepMind Control MetaWorld HumanoidBench Atari (ALE) ShadowHand
Benchmarks: DMC-Hard MetaWorld HumanoidBench-Medium HumanoidBench-Hard ShadowHand Atari MW+DMC
Metrics: normalized return success rate 95% bootstrapped confidence interval sample efficiency Shapley values
Baselines: SAC MH-SAC BRO SimBa SimBaV2 TD-MPC2 DreamerV3 PCGrad PaCo CTPG
Builds on: SAC BRO BroNet DrQ-epsilon categorical RL (C51)
Closest papers by meaning
- Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Representation Learning Enables Scalable Multitask Deep Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Bridging Environments and Language with Rendering Functions and Vision-Language Models (ICML 2024 · similarity 0.89)
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.88)
- Vision Language Models are In-Context Value Learners (ICLR 2025 · similarity 0.88)
- Stop Regressing: Training Value Functions via Classification for Scalable Deep RL (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.