Research map / Reinforcement learning / Offline RL
Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes consistency policies (consistency models as RL action generators) with Consistency-BC and Consistency-AC, showing comparable D4RL performance to diffusion policies at much lower computational cost across offline, offline-to-online, and online RL.
Problem
Diffusion policies are expressive for multi-modal action distributions but their iterative sampling is slow and backpropagation through many denoising steps is costly, which limits their use in RL, especially online RL.
Approach
Use a conditional consistency model as the policy representation (state as condition, action as generated variable), so actions are produced in very few denoising steps (N=2 by default), and train it with a consistency BC loss plus a Q-learning policy term in an actor-critic loop, with a k-dependent loss scaling to balance the two terms.
Key results
- Consistency-BC reaches an average of 69.7 on D4RL Gym versus 76.3 for Diffusion-BC, and Diffusion-BC is 1.43x? no; Consistency-BC cuts BC training time by about 43% (42.97% average reduction across 20 tasks).
- Consistency-AC averages 72.1 on offline D4RL tasks versus 80.3 for Diffusion-QL, while using N=2 denoising steps instead of N=5.
- In online RL from scratch Consistency-AC averages 70.4 versus 67.7 for Diffusion-QL, with lower wall-clock time (7.49h vs 8.36h average); offline-to-online averages 88.0 vs 94.2 for Diffusion-QL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Gaussian policy with consistency policy for behavior cloning | BC on D4RL · D4RL tasks | normalized score: outperforms it on $14/20$ tasks | Helped |
| use consistency policy instead of diffusion policy for behavior cloning | BC on D4RL · D4RL task suite average | training time per epoch: $86.08\pm3.16$ -> $49.09\pm2.34$ | Helped |
| use consistency policy instead of diffusion policy in actor-critic offline RL | Consistency-AC · D4RL total average | normalized score: 80.3 -> 72.1 | Hurt |
| use consistency policy instead of diffusion policy | offline-to-online RL on D4RL Gym · D4RL Gym offline-to-online average | normalized score: 94.2 -> 88.0 | Hurt |
| use consistency policy instead of diffusion policy | online RL on D4RL Gym · D4RL Gym online average | normalized score: 67.7 -> 70.4 | Helped |
| remove loss scaling (ablation) | Consistency-AC · D4RL offline RL tasks | normalized score: by $37.8\%$ on average | Hurt |
| add loss scaling (ablation) | Consistency-AC · D4RL total average | normalized score: 40.1 -> 64.5 | Helped |
| add Q-learning loss and policy regularization to behavior cloning (ablation) | Consistency-AC · D4RL total average | normalized score: 59.7 -> 64.5 | Helped |
| replace MLP with LN-Resnet parameterization (ablation) | Consistency-AC · D4RL tasks | normalized score: 64.5 -> 56.8 | Mixed |
| reduce denoising steps from 5 to 2 and use consistency model | actor-critic on hopper-medium-expert-v2 · hopper-medium-expert-v2 | average normalized score: $108.2\pm5.6$ ($112.3\pm0.2$) -> $102.4\pm3.0$ ($106.2\pm1.6$) | Hurt |
Surprising findings
- Consistency policy with N=2 steps matches Diffusion-QL with N=5 steps in offline performance, and removing the proposed loss scaling drops average Consistency-AC performance substantially.
- Consistency-AC is worse than Diffusion-QL in offline-to-online RL but outperforms it in online RL from scratch, which the authors attribute to diffusion policies being overly explorative.
Limitations
- Consistency-AC performs poorly on AntMaze tasks (average 70.1 vs 78.7 for Diffusion-QL), where sparse rewards make Q-learning the bottleneck.
- Consistency policy is slightly less expressive than diffusion policy in BC, and offline-to-online results are slightly lower than Diffusion-QL.
- Online wall-clock gains are smaller because environment simulation time dominates.
Experimental setup
Datasets: D4RL halfcheetah D4RL hopper D4RL walker2d D4RL AntMaze D4RL Adroit D4RL Kitchen
Benchmarks: D4RL Gym D4RL AntMaze D4RL Adroit D4RL Kitchen
Metrics: normalized score training time per epoch inference time per sample wall-clock time
Baselines: Diffusion-QL Diffusion-BC BC CQL IQL X-QL ARQ IDQL-A AWAC ACA SAC Diffuser
Builds on: consistency models Diffusion-QL TD3+BC Karras noise schedule
Closest papers by meaning
- Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization (NeurIPS 2024 · similarity 0.89)
- Score as Action: Fine Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning (ICML 2025 · similarity 0.88)
- Flow-Based Policy for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Efficient Online Reinforcement Learning for Diffusion Policy (ICML 2025 · similarity 0.87)
- Prioritized Generative Replay (ICLR 2025 · similarity 0.87)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.