Research map / Reinforcement learning / Policy gradient and actor-critic
CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CrossQ, a UTD=1 SAC variant that removes target networks and uses Batch Renormalization in critics (with concatenated current and next-state batches in one forward pass) plus wider critics, matching REDQ/DroQ sample efficiency at a fraction of compute.
Problem
Sample efficiency in deep RL for continuous control; state-of-the-art high-UTD methods (REDQ, DroQ) achieve it only with UTD=20 and high computational cost.
Approach
Removing target networks and using BatchNorm (Batch Renormalization) in the critic, with current and next-state batches concatenated in one forward pass so normalization statistics cover both distributions, lets a UTD=1 SAC-based method learn as fast as high-UTD methods without ensembles or bias-reduction schemes.
Key results
- CrossQ matches or surpasses REDQ and DroQ in sample efficiency on MuJoCo tasks, substantially exceeding all baselines on Humanoid and HumanoidStandup.
- CrossQ uses 20x fewer gradient steps than REDQ and DroQ and achieves roughly 4x faster wallclock time (e.g., HumanoidStandup 2.2h vs 8.7h for REDQ on RTX 3090).
- Removing target networks alone gives small gains, while removing target networks and adding BatchNorm gives the largest acceleration in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove target networks and add BatchNorm (ablation) | SAC · MuJoCo continuous-control environments | sample efficiency | Helped |
| remove target networks (ablation) | SAC with tanh critic activations · MuJoCo continuous-control environments | sample efficiency | Helped |
| add BatchNorm while retaining target networks (ablation) | SAC with tanh critic activations · MuJoCo continuous-control environments | sample efficiency | Mixed |
| reduce critic width to 256 (ablation) | CrossQ · MuJoCo continuous-control environments | sample efficiency | Hurt |
| remove BRN layers (ablation) | CrossQ · MuJoCo continuous-control environments | sample efficiency | Hurt |
| replace BatchNorm with LayerNorm (ablation) | CrossQ · MuJoCo continuous-control environments | sample efficiency | Hurt |
| widen critic layers to 2048 (ablation) | SAC · MuJoCo continuous-control environments | sample efficiency | No clear effect |
| add BRN while retaining target networks (ablation) | SAC · MuJoCo continuous-control environments | sample efficiency | Hurt |
| use BRN in the critic only instead of in both actor and critic (ablation) | CrossQ · MuJoCo continuous-control environments | sample efficiency | No clear effect |
| replace BRN with original BatchNorm (ablation) | CrossQ · MuJoCo continuous-control environments | training stability | Hurt |
| increase BRN moving-average momentum (ablation) | CrossQ · MuJoCo continuous-control environments | training stability | Helped |
| reduce Adam beta1 and use policy delay 3 (ablation) | CrossQ · MuJoCo continuous-control environments | sample efficiency | No clear effect |
Surprising findings
- Q estimation bias does not reliably predict learning performance; CrossQ performs well despite generally higher Q bias than REDQ.
- Naively adding BatchNorm while keeping target networks is unstable and detrimental, due to mismatched running statistics between live and target networks.
Limitations
- Evaluated only on state-based continuous control benchmarks, not visual observations or real robots.
- Wider critics (2048) increase computation cost relative to standard SAC.
- Theoretical understanding of BatchNorm use in off-policy RL is lacking; the bounded-activation trick used in ablations is empirical.
Experimental setup
Datasets: MuJoCo Hopper MuJoCo Walker2d MuJoCo Ant MuJoCo Humanoid MuJoCo HalfCheetah HumanoidStandup DeepMind Control Suite
Benchmarks: MuJoCo Hopper MuJoCo Walker2d MuJoCo Ant MuJoCo Humanoid MuJoCo HalfCheetah HumanoidStandup DeepMind Control Suite
Metrics: episodic return IQM return wallclock time normalized Q estimation bias
Baselines: REDQ DroQ SAC TD3
Builds on: Soft Actor-Critic REDQ DroQ TD3 Batch Renormalization
Closest papers by meaning
- Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization (NeurIPS 2025 · similarity 0.95)
- Sample-Efficiency in Multi-Batch Reinforcement Learning: The Need for Dimension-Dependent Adaptivity (ICLR 2024 · similarity 0.86)
- Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control (NeurIPS 2024 · similarity 0.86)
- Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers (NeurIPS 2024 · similarity 0.85)
- Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning (ICLR 2025 · similarity 0.85)
- Normalization and effective learning rates in reinforcement learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.