Research map / Reinforcement learning / Visual and representation RL
Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CP3ER, a consistency-policy actor-critic for visual RL that stabilizes Q-loss training with sample-based entropy regularization and prioritized proximal experience sampling, built on DrQ-v2.
Problem
Consistency policies trained with Q-loss in actor-critic frameworks become unstable in online RL, especially in visual RL with high-dimensional observations, as policy networks become dormant.
Approach
Measures policy-network dormant ratio to show Q-loss drives collapse; regularizes the consistency policy with a sample-based consistency loss on actions drawn from a proxy distribution, using a prioritized sampling of recent replay data as that proxy, and a mixture-of-Gaussians distributional critic.
Key results
- CP3ER achieves better sample efficiency and stability than DrQ-v2, ALIX, TACO on 8 medium-level DMC tasks and is reported as SOTA on 21 visual tasks across DMC and Meta-world.
- On 7 hard DMC tasks trained for 5M frames, CP3ER matches DrM on 4 tasks and outperforms it on 3 harder tasks without exploration strategies.
- On state-based DMC tasks, CP3ER averages 665.5 vs. 606.3 for CPQL (best prior baseline in the table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CP3ER instead of CPQL | State-based RL · Acrobot Swingup | return: 183.1 -> 362.9 | Helped |
| Use CP3ER instead of CPQL | State-based RL · Finger Turn Easy | return: 874.1 -> 867.45 | Hurt |
| Use CP3ER instead of CPQL | State-based RL · Finger Turn Hard | return: 864.6 -> 912.65 | Helped |
| Use CP3ER instead of CPQL | State-based RL · Hopper Hop | return: 130.1 -> 299.3 | Helped |
| Use CP3ER instead of CPQL | State-based RL · Hopper Stand | return: 902.1 -> 904.1 | Helped |
| Use CP3ER instead of CPQL | State-based RL · Walker Run | return: 683.8 -> 646.5 | Hurt |
| Use CP3ER instead of Consistency-AC | Consistency policy · Halfcheetah-m | return: 6725.2$\pm$944.4 -> 10699.1$\pm$1054.7 | Helped |
| Use CP3ER instead of Diffusion-QL | Online RL · Hopper-m | return: 3675.5$\pm$47.7 -> 3309.6$\pm$133.2 | Hurt |
| Use CP3ER instead of Consistency-AC | Consistency policy · Walker2d-m | return: 3790.9$\pm$1677.5 -> 5201.0$\pm$111.3 | Helped |
| Add prioritized proximal experience regularization (ablation) | Consistency policy · Visual control tasks (2 tasks) | performance | Helped |
| Use prioritized proximal experience sampling rather than a uniform proxy distribution (ablation) | Consistency policy · Visual control tasks (2 tasks) | sample efficiency | Helped |
| Use prioritized proximal experience sampling rather than uniform replay-buffer behavior sampling (ablation) | Consistency policy · Visual control tasks (2 tasks) | sample efficiency | Helped |
Surprising findings
- Non-stationarity of online replay data does not significantly change consistency-model dormant ratios compared with offline training.
- The authors could not reproduce DrM's reported Meta-world hard-task performance using official code.
Limitations
- Policy diversity depends on multimodal actions in the replay buffer, which diminish over training, risking loss of exploration.
- Lacks theoretical analysis of policy improvement and convergence.
- Auxiliary representation losses not yet combined with consistency policy.
Experimental setup
Datasets: DeepMind control suite Meta-world MuJoCo Halfcheetah MuJoCo Walker2d D4RL
Benchmarks: DeepMind control suite (medium and hard tasks) Meta-world MuJoCo Halfcheetah-m MuJoCo Hopper-m MuJoCo Walker2d-m state-based DMC tasks
Metrics: episode return success rate mean IQM median optimality gap performance profiles probability of improvement dormant ratio
Baselines: DrQ-v2 ALIX TACO DrM Consistency-AC Diffusion-QL CPQL DreamerV3
Builds on: consistency models Consistency-AC DrQ-v2 dormant neuron phenomenon DrM
Closest papers by meaning
- Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning (ICLR 2024 · similarity 0.89)
- What Effects the Generalization in Visual Reinforcement Learning: Policy Consistency with Truncated Return Prediction (AAAI 2024 · similarity 0.89)
- The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations (NeurIPS 2024 · similarity 0.88)
- VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Seek Commonality but Preserve Differences: Dissected Dynamics Modeling for Multi-modal Visual RL (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.