Research map / Reinforcement learning / MDPs and planning
Model-based Reinforcement Learning for Parameterized Action Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DLPA, the first model-based RL method for parameterized action MDPs, combining a parameterized-action-conditioned dynamics model trained with H-step loss and a PAMDP-specific MPPI planner, with theoretical Lipschitz regret bounds.
Problem
Sample-inefficient model-free RL for parameterized action Markov decision processes (hybrid discrete-continuous action spaces), where no prior model-based method existed.
Approach
Learns a dynamics model conditioned on discrete action and its continuous parameter, trained with a multi-step (H-step) prediction loss that backpropagates through imagined rollouts, and plans with an MPPI variant that keeps a separate continuous-parameter distribution per discrete action.
Key results
- DLPA reaches better asymptotic performance than all baselines on 6 of 8 tasks, e.g. Hard Move (10) 6.35 vs HyAR -7.05, and Catch Point 7.56 vs HyAR 5.20.
- DLPA reports on average about 30x higher sample efficiency than the best model-free method per scenario.
- Removing separate reward predictors drops Catch Point from 7.56 to -17.65 and Hard Move (4) from 6.29 to -37.38.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DLPA instead of HyAR | PAMDP agents · Platform | return: 0.92\pm 0.05 vs 0.98\pm0.08 | Hurt |
| use DLPA instead of HyAR | PAMDP agents · Goal | return: 28.75\pm6.91 vs 34.23\pm3.71 | Hurt |
| use DLPA instead of HyAR | PAMDP agents · Hard Goal | return: 28.38\pm2.88 vs 26.41\pm3.59 | Helped |
| use DLPA instead of PDQN | PAMDP agents · Catch Point | return: 7.56\pm4.86 vs 6.64\pm2.52 | Helped |
| use DLPA instead of HyAR | PAMDP agents · Hard Move (4) | return: 6.29\pm5.74 vs 6.09\pm 1.67 | Helped |
| use DLPA instead of HyAR | PAMDP agents · Hard Move (6) | return: 8.48\pm5.45 vs 6.33\pm2.12 | Helped |
| use DLPA instead of HyAR | PAMDP agents · Hard Move (8) | return: 7.80\pm6.27 vs -0.88\pm3.83 | Helped |
| use DLPA instead of HyAR | PAMDP agents · Hard Move (10) | return: 6.35\pm9.97 vs -7.05\pm3.74 | Helped |
| add separate reward predictors conditioned on termination rather than one unified predictor (ablation) | DLPA · Catch Point | return: -17.65\pm7.75 -> 7.56\pm4.86 | Helped |
| add separate reward predictors conditioned on termination rather than one unified predictor (ablation) | DLPA · Hard Move (4) | return: -37.38\pm8.62 -> 6.29\pm5.74 | Helped |
| add separate reward predictors conditioned on termination rather than one unified predictor (ablation) | DLPA · Hard Move (6) | return: -39.66\pm10.60 -> 8.48\pm5.45 | Helped |
| add separate reward predictors conditioned on termination rather than one unified predictor (ablation) | DLPA · Hard Move (8) | return: -40.17\pm11.82 -> 7.80\pm6.27 | Helped |
Surprising findings
- Without learning an action embedding, DLPA outperforms HyAR on large discrete action spaces (Hard Move with up to 1024 discrete actions), with the gap widening as the action space grows.
- Standard PAMDP model-free baselines fail entirely on Hard Move (n>=6) without a learned action embedding, while DLPA succeeds by sampling the raw action space.
Limitations
- Planning cost grows substantially with the number of discrete actions (2.85 s per step at 2^10 actions).
- Theoretical bounds rely on Lipschitz assumptions and a bounded-error learned model, and the Gaussian Wasserstein bound in the lemma ignores the covariance term.
- Evaluation is limited to standard PAMDP benchmarks with short horizons; the longer-horizon HFO results are only in the appendix.
Experimental setup
Datasets: Platform Goal Hard Goal Catch Point Hard Move (4) Hard Move (6) Hard Move (8) Hard Move (10)
Benchmarks: Platform Goal Hard Goal Catch Point Hard Move (4) Hard Move (6) Hard Move (8) Hard Move (10)
Metrics: asymptotic return (mean of last ten steps) sample efficiency planning clock time
Baselines: HyAR P-DQN PATD3 PADDPG HPPO
Builds on: Model Predictive Path Integral HyAR P-DQN TD3 Cross-Entropy Method
Closest papers by meaning
- Reinforcement Learning with Parameterized Actions (AAAI 2016 · similarity 0.91)
- Context-Sensitive Abstractions for Reinforcement Learning with Parameterized Actions (AAAI 2026 · similarity 0.88)
- Predictable MDP Abstraction for Unsupervised Model-Based RL (ICML 2023 · similarity 0.87)
- Meta-learning Parameterized Skills (ICML 2023 · similarity 0.87)
- Deterministic Policy Optimization by Combining Pathwise and Score Function Estimators for Discrete Action Spaces (AAAI 2018 · similarity 0.87)
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.