Research map / Reinforcement learning / Policy gradient and actor-critic
Trust-Region Twisted Policy Improvement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tailors variational sequential Monte-Carlo planning for RL (TRT-SMC) via trust-region twisted proposals, ancestor-level message-passing policy inference, search-based value targets, and revived resampling for terminal states.
Problem
Particle-filter (SMC) planners for RL suffer from path degeneracy, wasted compute on absorbing/terminal states, and inefficient use of the particle budget, so they do not deliver good local policy improvement at the root compared with MCTS.
Approach
Combine exponentially twisted proposals constrained by an adaptive KL trust-region to the prior policy, online ancestor-level soft-value accumulation (avoiding path degeneracy), SMC-based search values as TD targets, and resampling back to the last non-terminal state.
Key results
- On Brax Ant, search-based SMC value targets (V_SMC) reached a mean final return of 7457.1 versus 6911.8 for the 1-step value V_theta.
- On Jumanji Snake, revived resampling gave 45.7 mean return versus 44.7 for baseline, with overlapping confidence intervals (not significant).
- Constrained (alpha=0.1) twisted proposals improve performance over the prior proposal (alpha=0) at low particle budgets, with the gap shrinking at higher budgets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TRT-SMC instead of variational SMC | SMC planning for RL · Brax continuous-control tasks and Jumanji discrete environments | evaluation return | Helped |
| increase planning budget for Gumbel MCTS | Gumbel AlphaZero · Jumanji discrete environments | runtime | Hurt |
| use a constrained, Q-twisted proposal with α = 0.1 instead of the prior proposal with α = 0 (ablation) | TRT-SMC · Jumanji environments at low particle budgets | final test performance | Helped |
| mix the prior policy with maximizing the predicted Qθπ instead of relying completely on either (ablation) | TRT-SMC · All tested environments | normalized final test performance | Helped |
| replace Dirac-mixture policy inference with message-passing backups (ablation) | TRT-SMC · Jumanji environments as planner depth increases | final performance | Helped |
| use SMC-based values instead of Vθ for outer learning targets (ablation) | TRT-SMC · Brax Ant | final expected episode return: 6911.8 -> 7457.1 | Helped |
| use mixed Vθ and VSMC values instead of Vθ for outer learning targets (ablation) | TRT-SMC · Brax Ant | final expected episode return: 6911.8 -> 7214.7 | Helped |
| use revived resampling instead of baseline resampling (ablation) | TRT-SMC · Jumanji Snake | final expected episode return: 44.7 -> 45.7 | No clear effect |
Surprising findings
- Revived resampling did not significantly differ from the baseline on Jumanji Snake.
- Gumbel MCTS scaled poorly in wallclock time compared with SMC planners.
Limitations
- Experiments assume access to a highly accurate simulator; planner transitions not counted in main sample-efficiency results.
- Revived resampling showed no significant benefit in the tested setting.
- Only a few environments (two Jumanji, two Brax) tested.
Experimental setup
Datasets: Jumanji Snake Jumanji RubiksCube (partly scrambled) Brax Ant Brax Halfcheetah
Benchmarks: Jumanji Snake Jumanji RubiksCube Brax Ant Brax Halfcheetah
Metrics: average offline episode return over 128 episodes normalized return (min-max) 99% BCa bootstrap confidence intervals wallclock runtime estimate
Baselines: variational SMC planner Gumbel AlphaZero (Gumbel MCTS) PPO A2C SAC
Builds on: variational sequential Monte-Carlo planning (Dhir et al. style SMC for RL) Gumbel AlphaZero / MCTS MPO Retrace(lambda) MuZero
Closest papers by meaning
- Twice Sequential Monte Carlo for Tree Search (ICML 2026 · similarity 0.90)
- Online Robust Reinforcement Learning Through Monte-Carlo Planning (ICML 2025 · similarity 0.87)
- MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planning (NeurIPS 2025 · similarity 0.85)
- Monte Carlo Tree Diffusion for System 2 Planning (ICML 2025 · similarity 0.85)
- Monte Carlo Tree Search in the Presence of Transition Uncertainty (AAAI 2024 · similarity 0.85)
- Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.