Research map / Reinforcement learning / Policy gradient and actor-critic
Hyperspherical Normalization for Scalable Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SimbaV2, an RL architecture that stabilizes norms via hyperspherical feature/weight normalization plus distributional critic with reward scaling, enabling SAC to scale with model size and UTD ratio on 57 continuous control tasks.
Problem
Scaling model size and compute (via update-to-data ratio) fails in deep RL because non-stationary data causes overfitting and unstable optimization driven by uncontrolled weight, feature, and gradient norm growth.
Approach
Constrain weights and features to the unit hypersphere (l2-normalization, weight projection after each update, LERP as retraction-like residual), add a shift-based input projection, and use a categorical critic with running reward scaling, all within SAC, removing weight decay and periodic reinitialization.
Key results
- SimbaV2 with UTD=2 reaches 0.892 average normalized score across 57 tasks at low UTD, exceeding the previous best of 0.780 (Simba).
- Increasing SimbaV2 UTD from 2 to 8 raises the average score from 0.892 to 0.911, outperforming BRO at UTD=10 (0.807).
- On offline D4RL MuJoCo, SimbaV2+BC averages 86.7, competitive with baselines and significantly better on Hopper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SimbaV2 instead of Simba | SAC · All (57), low UTD (≤ 2) | normalized performance: 0.780 ± 0.028 -> 0.892 ± 0.032 | Helped |
| Increase UTD from 2 to 8 | SimbaV2 · All (57) | normalized performance: 0.892 to 0.911 | Helped |
| Add behavioral cloning loss | SimbaV2 · D4RL Hopper | normalized score | Helped |
| Add input L2 normalization (ablation) | SimbaV2 · All (57), design study | normalized performance: 0.809 ± 0.059 -> 0.892 ± 0.032 | Helped |
| Add input shifting to preserve magnitude (ablation) | SimbaV2 · All (57), design study | normalized performance: 0.810 ± 0.061 -> 0.892 ± 0.032 | Helped |
| Change c_shift from 1 to 3 (ablation) | SimbaV2 · All (57), design study | normalized performance: 0.888 ± 0.058 -> 0.892 ± 0.032 | No clear effect |
| Replace shift projection with resize projection (ablation) | SimbaV2 · All (57), design study | normalized performance: 0.884 ± 0.058 -> 0.892 ± 0.032 | No clear effect |
| Use categorical loss instead of MSE loss (ablation) | SimbaV2 · All (57), design study | normalized performance: 0.846 ± 0.062 -> 0.892 ± 0.032 | Helped |
| Add reward scaling (ablation) | SimbaV2 · All (57), design study | normalized performance: 0.852 ± 0.042 -> 0.892 ± 0.032 | Helped |
| Add reward bounding (ablation) | SimbaV2 · All (57), design study | normalized performance: 0.868 ± 0.051 -> 0.892 ± 0.032 | Helped |
| Use soft targets instead of hard targets (ablation) | SimbaV2 · All (57), design study | normalized performance: 0.890 ± 0.051 -> 0.892 ± 0.032 | No clear effect |
| Use learning-rate decay (ablation) | SimbaV2 · All (57), design study | normalized performance: 0.863 ± 0.064 -> 0.892 ± 0.032 | Helped |
Surprising findings
- Periodic weight reinitialization slightly degrades SimbaV2's performance, unlike prior methods where it is required for compute scaling.
- Applying the distributional critic and reward scaling to plain Simba alone fails to scale at higher UTD ratios, showing the hyperspherical normalization is essential.
Limitations
- Scaling benefits saturate on DMC-Hard at depth L=4, attributed to task complexity.
- Reinitialization and high-UTD gains are evaluated only on the paper's benchmarks, and the approach is untested on visual RL or large language model RL.
- Reward bounding is needed for easier tasks such as cartpole; without it performance collapses.
Experimental setup
Datasets: MuJoCo DMC Suite MyoSuite HumanoidBench D4RL MuJoCo
Benchmarks: MuJoCo DMC Suite MyoSuite HumanoidBench D4RL MuJoCo
Metrics: normalized return success rate 95% confidence interval Welch's t-test
Baselines: SAC TD-MPC2 BRO Simba MR.Q TD7 CrossQ DreamerV3
Builds on: Simba Soft Actor-Critic BRO CrossQ
Closest papers by meaning
- SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning (ICLR 2025 · similarity 0.89)
- Bigger, Regularized, Optimistic: scaling for compute and sample efficient continuous control (NeurIPS 2024 · similarity 0.86)
- Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Representation Learning Enables Scalable Multitask Deep Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
- Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization (NeurIPS 2025 · similarity 0.86)
- Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.