Research map / Reinforcement learning / Visual and representation RL
DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DrM, a visual model-free RL method for continuous control that periodically perturbs weights and uses dormant-ratio-guided exploration and exploitation schedules to minimize the dormant ratio of the policy network.
Problem
Visual RL agents for continuous control suffer from sustained motor inactivity early in training, poor sample efficiency on high-DoF tasks (Dog, Manipulator, Adroit), and broken random seeds.
Approach
Uses the dormant ratio (fraction of inactive neurons in the policy network) as an intrinsic activity signal to drive three mechanisms: periodic soft weight perturbation when dormant ratio is high, a dormant-ratio-based exploration noise scheduler, and a dormant-ratio-based expectile exploitation parameter lambda.
Key results
- DrM is the first documented model-free visual RL method to solve both Dog and Manipulator domains in DeepMind Control Suite from pixels, using 4 seeds per task with no broken seeds reported.
- DrM requires about 70%, 45%, and 60% fewer samples to match baseline peak performance on DMC, MetaWorld, and Adroit respectively, per the paper's claims.
- In the Adroit ablation, removing dormant-ratio-guided exploitation lowers final success by nearly 20%, while removing perturbation or awaken exploration lowers it by close to 40%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DrM instead of the baseline methods | visual RL agents · DeepMind Control Suite | samples to match peak asymptotic performance: 70% fewer samples | Helped |
| Use DrM instead of the baseline methods | visual RL agents · MetaWorld | samples to match peak asymptotic performance: 45% fewer samples | Helped |
| Use DrM instead of the baseline methods | visual RL agents · Adroit | samples to match peak asymptotic performance: 60% fewer samples | Helped |
| Use DrM instead of the best-performing baseline | visual RL agents · DeepMind Control Suite | asymptotic performance: 65% | Helped |
| Use DrM instead of the best-performing baseline | visual RL agents · MetaWorld | asymptotic performance: 35% | Helped |
| Use DrM instead of the best-performing baseline | visual RL agents · Adroit | asymptotic performance: 75% | Helped |
| Add dormant-ratio-guided exploitation (ablation) | DrM · Adroit | final success rate: nearly 20% | Helped |
| Add dormant-ratio-guided perturbation (ablation) | DrM · Adroit | final success rate: close to 40% | Helped |
| Add awaken exploration (ablation) | DrM · Adroit | final success rate: close to 40% | Helped |
| Remove dormant-ratio-guided perturbation (ablation) | DrM · Adroit | success rate: just about 40% | Hurt |
| Replace dormant-ratio-guided parameters with fixed parameters and a fixed linear exploration schedule (ablation) | DrM · Adroit | performance | Hurt |
| Replace adaptive dormant-ratio exploration noise with maximal exploration noise before awakening (ablation) | DrM · Adroit | performance | Hurt |
Surprising findings
- Baselines (DrQ-v2, ALIX, TACO) show sudden performance collapse for some seeds on dog tasks, while DrM does not.
- DrQ-v2 achieves 0% success on Adroit tasks, and adding RND only partially helps, suggesting intrinsic exploration alone is insufficient.
Limitations
- Evaluated only on continuous control; discrete-action extension is proposed but not tested.
- Relies on several hand-set hyperparameters (threshold, temperature, perturb factors) and some tasks use domain-specific settings.
- Main experiments use only 4 random seeds per task, though an appendix reports 10 runs on some MetaWorld tasks.
Experimental setup
Datasets: DeepMind Control Suite MetaWorld Adroit
Benchmarks: DeepMind Control Suite MetaWorld Adroit
Metrics: success rate episode reward dormant ratio frames per second
Baselines: DrQ-v2 ALIX TACO RND ReDo
Builds on: DrQ-v2 ReDo shrink-and-perturb expectile regression (IQL-style) NoisyNet
Closest papers by meaning
- DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors (NeurIPS 2024 · similarity 0.89)
- Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning (ICML 2025 · similarity 0.88)
- Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages (ICLR 2024 · similarity 0.88)
- The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Harnessing Density Ratios for Online Reinforcement Learning (ICLR 2024 · similarity 0.88)
- A Forget-and-Grow Strategy for Deep Reinforcement Learning Scaling in Continuous Control (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.