Research map / Reinforcement learning / Policy gradient and actor-critic
Iterative Gumbel Planning for Continuous Control
Shaohuai Liu, Weirui Ye, Yilun Du, Le Xie
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Iterative Gumbel Planning, a search-based policy improvement method for continuous control that addresses the mismatch between finite-candidate search and parametric continuous policies.
Problem
Search-based planning (AlphaZero/MuZero-style) works well for discrete actions, but in continuous control search over a finite candidate set returns a discrete distribution, which mismatches the parametric continuous policy class.
Approach
Extends Gumbel-based policy improvement to continuous action spaces by iteratively planning over candidate actions so the search output can be distilled into a parametric continuous policy; the abstract is truncated, so the precise mechanism is not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Gaussian policy with iteratively refined factorized categorical policy | EfficientZero-Multitask · DMC-Hard dog-run | max return: 405.874 -> 613.257 | Improved |
| replace Gaussian policy with iteratively refined factorized categorical policy | EfficientZero-Multitask · DMC-Hard humanoid-walk | max return: 779.404 -> 911.542 | Improved |
| replace Gaussian policy with factorized policy | EfficientZero-Multitask · HumanoidBench h1hand-slide | max return: 666.507 -> 870.521 | Improved |
| replace Gaussian policy with factorized policy | EfficientZero-Multitask · HumanoidBench h1hand-reach, crawl, stair, sit_hard | max return: reach 5820.811 -> 5114.852 | Hurt |
| use factorized categorical policy instead of refinement-matched Gaussian proposal (ablation) | EZ-IGP · DMC humanoid-stand | return: 763.8 -> 903.7 | Improved |
| distill search-score-weighted mixture of candidates instead of winner action into Gaussian (ablation) | Gaussian policy · DMC humanoid | policy sigma and return: final sigma 2.67 | No clear effect |
| increase bins V from 11 to 31 (ablation) | EZ-IGP · HB-Hard normalized score | normalized score: 0.622 -> 0.743 | Improved |
| increase bins V from 31 to 41 (ablation) | EZ-IGP · HB-Hard normalized score | normalized score: 0.743 -> 0.690 | Hurt |
| use two refinement rounds instead of one (ablation) | EZ-IGP · HB-Hard normalized score | normalized score: 0.667 -> 0.743 | Improved |
| increase refinement rounds from 2 to 4 (ablation) | EZ-IGP · HB-Hard normalized score | normalized score: 0.743 -> 0.689 | Hurt |
| reduce candidates/simulations from 8/16 to 2/4 (ablation) | EZ-IGP · DMC humanoid average | average return: 728.243 -> 296.748 | Hurt |
| use factorized policy instead of Gaussian | EZ-IGP · evaluation action smoothness | temporal action variance | Improved |
Experimental setup
Builds on: AlphaZero MuZero Gumbel MuZero
Closest papers by meaning
- Policy improvement by planning with Gumbel (ICLR 2022 · similarity 0.87)
- Graph Sparse Sampling: Breaking the Curse of the Horizon in Continuous MDP Planning (NeurIPS 2026 · similarity 0.87)
- Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Augmented Lagrangian Method for Last-Iterate Convergence for Constrained MDPs (NeurIPS 2026 · similarity 0.85)
- Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.85)
- Deterministic Policy Gradient Primal-Dual Methods for Continuous-Space Constrained MDPs (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
