Research map / Reinforcement learning / Visual and representation RL

1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities

NeurIPS 2025 · Reinforcement learning › Visual and representation RL · approach: Subgoal generation for hierarchical RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that scaling the depth of contrastive RL (with residual blocks, layer norm, and Swish) to up to 1024 layers yields 2-50x gains on goal-reaching tasks, with qualitatively new behaviors.

Problem

Whether self-supervised goal-conditioned RL can scale like language and vision models, given that RL networks are typically only 2-5 layers deep and deeper networks often fail to help.

Approach

Combine contrastive RL with residual connections, layer normalization, and Swish activations, plus GPU-accelerated massively parallel environments, and scale network depth (actor and critic encoders) up to 1024 layers instead of width alone.

Key results

• Scaling CRL depth from 4 to 64 gives 2.5x-5.7x gains on manipulation tasks and 25x-63x on Ant U4/U5-Maze, with up to ~1051x on Humanoid Big Maze (Table 1).
• Scaled CRL outperforms SAC, SAC+HER, TD3+HER, GCSL, and GCBC in 8 of 10 environments.
• On Humanoid U-Maze, performance continues to improve at depths 256 and 1024.

Measured effects reported in the paper

Surprising findings

• Depth scaling is not effective for SAC, SAC+HER, and TD3+HER, which saturate at depth 4, suggesting the self-supervised CRL objective is key.
• Scaling depth in offline goal-conditioned RL (OGBench) yielded little or negative gains, with depth-4 often best.

Limitations

• Scaling depth increases compute and wall-clock cost.
• Actor loss exploded at 1024 layers, so the actor was held at 512 layers for that setting.
• Offline goal-conditioned RL did not benefit from depth scaling in the reported experiments.

Experimental setup

Datasets: JaxGCRL environments (Brax/MJX) OGBench

Benchmarks: JaxGCRL (10 locomotion, navigation, manipulation tasks) OGBench antmaze-medium-stitch

Metrics: time steps near goal out of 1000 (success) steps to reach success threshold wall-clock time

Baselines: CRL (original, depth 4) SAC SAC+HER TD3+HER GCBC GCSL QRL CMD-1

Builds on: contrastive RL JaxGCRL Brax MJX ResNet Simba-v2

Closest papers by meaning

• Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data (ICLR 2024 · similarity 0.91)
• Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.88)
• METRA: Scalable Unsupervised RL with Metric-Aware Abstraction (ICLR 2024 · similarity 0.88)
• Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.87)
• Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.87)
• Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning (ICML 2025 · similarity 0.87)