Research map / Reinforcement learning / Policy gradient and actor-critic
Langevin Policy for Safe Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Langevin Policy, a sampling-based policy for safe RL that runs Langevin dynamics on a CPO/FOCOPS-style energy, and Langevin Actor-Critic (LAC), which distills the Langevin sampler into a Gaussian generator trained with short-run informative-initialized MCMC.
Problem
Safe RL is dominated by optimization-based policy methods; it is unclear whether sampling-based (Langevin MCMC) policies can achieve good reward and constraint satisfaction, and naive Langevin sampling per action is too slow.
Approach
Treat the CPO/FOCOPS optimal policy as an energy-based model and sample actions with Langevin dynamics; accelerate by initializing the chain from a learned Gaussian generator (informative initialization) with a short fixed number of steps, and train the generator by maximum-likelihood-like loss on Langevin outputs, with clipped double Q-learning for reward and cost critics.
Key results
- On MuJoCo tasks after 2.5M steps, LAC reaches 1940.71 reward on Ant-v3 with cost 103.04 (limit 103.12) and 5023.27 reward on Humanoid-v3 with cost 20.12 (limit 20.14), outperforming CPO, CUP, FOCOPS, PPO-L and TRPO-L.
- On Safety Gym PointGoal1, LAC obtains reward 14.20 with cost 22.21 under a 25 cost limit, while CarGoal1 reward is 24.01 with cost 22.15.
- Ablation on PointGoal and CarGoal shows noninformative initialization degrades LAC (PointGoal reward 5.58 vs 14.20), indicating informative initialization matters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use noninformative instead of informative initialization (ablation) | LAC · PointGoal and CarGoal | Reward: 14.20 ± 3.35 vs 5.58 ± 1.65; 24.01 ± 2.21 vs 13.25 ± 7.92 | Hurt |
| vary the number of Langevin steps (ablation) | LAC · PointGoal and CarGoal | Reward: 14.20 ± 3.35 vs 15.33 ± 2.03; 24.01 ± 2.21 vs 23.00 ± 1.33 | Mixed |
| add policy KL-regularization (ablation) | LAC · PointGoal and CarGoal | Reward: 14.20 ± 3.35 vs 13.63 ± 1.36; 24.01 ± 2.21 vs 19.75 ± 5.07 | Hurt |
| add policy KL-regularization and train for 10 million steps (ablation) | LAC · PointGoal and CarGoal | Reward: 14.20 ± 3.35 vs 14.71 ± 0.70; 24.01 ± 2.21 vs 21.85 ± 4.47 | Mixed |
| use LAC instead of FOCOPS | safe RL on MuJoCo tasks · Ant-v3 and Humanoid-v3 | Reward: 1940.71 ± 16.31 vs 1803.86 ± 42.90; 5023.27 ± 107.99 vs 3196.58 ± 909.07 | Helped |
| use LAC instead of PPO-L | safe RL on MuJoCo tasks · Swimmer-v3 | Reward: 34.56 ± 4.94 vs 49.07 ± 3.33 | Hurt |
| use LAC instead of FOCOPS | safe RL on MuJoCo tasks · HumanoidCircle-v0 | Reward: 925.03 ± 111.95 vs 862.92 ± 67.24 | Helped |
| use LAC instead of CPO | Safety Gym tasks · PointGoal, PointButton, and CarButton | Reward: 14.20 ± 3.35 vs 23.16 ± 2.09; 2.45 ± 1.36 vs 18.42 ± 3.04; 1.50 ± 1.44 vs 10.16 ± 2.54 | Hurt |
| use LAC instead of CPO | safe RL on Safety Gym tasks · CarGoal | Reward: 24.01 ± 2.21 vs 25.24 ± 4.80 | Hurt |
Surprising findings
- Noninformative initialization needed T=500 steps to reach its best performance, while informative initialization works with T as low as 50-100.
Limitations
- Gaussian generator limits expressiveness of the approximated Langevin policy.
- Generator acts as intermediary, possibly introducing approximation error versus the true Langevin sampler.
- Requires Q-value gradients as energy score, which suffer from estimation bias.
Experimental setup
Datasets: MuJoCo Ant-v3 Swimmer-v3 Humanoid-v3 HumanoidCircle-v0 Safety Gym PointGoal1 PointButton1 CarGoal1 CarButton1
Benchmarks: MuJoCo Ant-v3 Swimmer-v3 Humanoid-v3 HumanoidCircle-v0 Safety Gym PointGoal1 PointButton1 CarGoal1 CarButton1
Metrics: average episodic reward average episodic cost
Baselines: CPO FOCOPS CUP PCPO P3O RESPO PPO-L TRPO-L
Builds on: CPO FOCOPS TD3 CUP
Closest papers by meaning
- Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning (ICLR 2025 · similarity 0.89)
- Mean Field Langevin Actor-Critic: Faster Convergence and Global Optimality beyond Lazy Learning (ICML 2024 · similarity 0.87)
- A Provable Approach for End-to-End Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Sampling-Based Safe Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
- Long-Term Safe Reinforcement Learning with Binary Feedback (AAAI 2024 · similarity 0.86)
- Probabilistic Shielding for Safe Reinforcement Learning (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.