Research map / Reinforcement learning / Visual and representation RL
Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies architecture, normalization, cold initialization, batch size and random-cropping augmentation to stabilize offline contrastive RL, yielding "stable contrastive RL" that solves image-based real-world WidowX manipulation from goal images.
Problem
Self-supervised goal-conditioned RL from offline image data is hard to deploy on real robots; prior contrastive RL is unstable and underperforms on image-based manipulation tasks.
Approach
Identify design decisions (3-layer CNN encoder with wide (1024,4) MLP, layer normalization, cold initialization of final layers, random-crop augmentation, large batch size 2048) that stabilize contrastive RL and let it solve real-world goal-image manipulation without auxiliary representation losses.
Key results
- Stable contrastive RL improves success on the drawer task from 31% to 84% (about +45% over prior contrastive RL implementations, and about 2x alternative goal-conditioned methods).
- Stable contrastive RL reaches 60% success on the real-world pick & place spoon and push can tasks, where GC-IQL and GCBC fail.
- Stable contrastive RL achieves 33% higher relative average success than the strongest baseline on Walker and Quadruped locomotion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a 3-layer CNN instead of ResNet 18 (ablation) | Stable contrastive RL · Simulated manipulation, drawer | success rate: 28% -> 81% | Helped |
| initialize final-layer weights with Unif[-10^{-12}, 10^{-12}] instead of Unif[-10^{-4}, 10^{-4}] (ablation) | Stable contrastive RL · Simulated manipulation, drawer | success rate: 33% -> 82% | Helped |
| replace the 3-layer CNN with ResNet 18 and use a deeper, wider MLP (ablation) | Stable contrastive RL · Push block, open drawer | success rate: 2.75× | Hurt |
| remove layer normalization (ablation) | Stable contrastive RL · Push block, open drawer | success rate: 24.5% | Hurt |
| use ImageNet-pretrained weights instead of random initialization (ablation) | Stable contrastive RL with ResNet-18 · Drawer | success rate | Helped |
| use ImageNet-pretrained weights instead of random initialization (ablation) | Stable contrastive RL with ResNet-18 · Push block, open drawer | success rate | No clear effect |
| increase offline dataset size (ablation) | Stable contrastive RL · Drawer | binary future prediction accuracy: 84% -> 98% | Helped |
| increase offline dataset size (ablation) | Stable contrastive RL · Drawer | success rate: 31% -> 84% | Helped |
| use a VAE representation as input to the actor and critic instead of learning representations end-to-end (ablation) | Contrastive RL · Push block, open drawer | success rate: 53% | Hurt |
| increase MLP depth from 2 to 4 or 6 layers, with 1024 units (ablation) | Stable contrastive RL · Push block, open drawer | success rate | No clear effect |
| use stable contrastive RL instead of GC-IQL or GCBC | Offline goal-conditioned policies · Real-world manipulation, reach eggplant | success rate | No clear effect |
| use stable contrastive RL instead of GC-IQL or GCBC | Offline goal-conditioned policies · Real-world manipulation, pick & place spoon and push can | success rate: 60% | Helped |
Surprising findings
- A 3-layer CNN outperforms a ResNet 18 encoder by 2.89x (81% vs 28%), which the authors attribute to ResNet overfitting on the offline data.
- Cold initialization with Unif[-1e-12, 1e-12] on final layers boosts performance 2.49x over Unif[-1e-4, 1e-4].
Limitations
- Scale of models and datasets is tiny compared with NLP and CV, so scaling benefits are only suggestive.
- Stable contrastive RL underperforms GC-IQL and GCBC on push block, close drawer, possibly due to partial observability.
- Real-world evaluation covers only three tasks, with 10 rollouts each.
Experimental setup
Datasets: Bridge Data simulated manipulation offline dataset (~250K transitions) locomotion replay buffer from LEXA (100K transitions)
Benchmarks: simulated manipulation suite (drawer, push block open drawer, push block close drawer, pick & place table, pick & place Walker Quadruped real-world WidowX 250 tasks (reach eggplant, pick & place spoon, push can)
Metrics: success rate binary future prediction accuracy
Baselines: contrastive RL GCBC GC-IQL GoFar WGCSL PTP VIP-GCBC R3M-GCBC
Builds on: contrastive RL PTP IQL C-Learning NCE
Closest papers by meaning
- 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities (NeurIPS 2025 · similarity 0.91)
- SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes (AAAI 2025 · similarity 0.88)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.88)
- Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.