Research map / Reinforcement learning / Visual and representation RL
Improving planning and MBRL with temporally-extended actions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes temporally-extended actions in which the planner optimizes both primitive actions and their durations, with a bandit-selected maximum duration, for faster planning and MBRL with learned temporally-extended dynamics models.
Problem
Discrete-time simulation of continuous systems needs a small timestep, forcing long planning horizons that make shooting-based planners (CEM/MPPI) expensive and inaccurate, and making learned models suffer compounding error in MBRL.
Approach
The planner outputs an action plus its duration as an extra optimization variable, so planning operates over temporally-extended actions with a shallow decision horizon while covering a deep primitive horizon; a learned temporally-extended dynamics model is used in MBRL, and the maximum duration is chosen via a non-stationary UCB multi-armed bandit over exponentially spaced candidates.
Key results
- On IPC multi-hill Mountain Car, the temporally-extended agent (D_TE=12) solves all 5 instances with success 1.00 across instances (except 0.80 on instance 4), while the standard agent (D=175) solves only instance 1.
- On the u-shaped Dubins Car map, the temporally-extended agent reaches success 1.0 with horizon 75 and 10,000 samples, while the standard agent needs horizon 1000; the standard agent fails with 100 dummy action variables due to memory.
- In MBRL, the temporally-extended agents are faster to train (e.g., Half Cheetah ~5 h vs 45 h for standard; Ant 4 h vs 41 h) with comparable or better rewards in most environments, but not in Reacher and Pusher.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use temporally-extended actions instead of primitive actions | CEM planning on IPC multi-hill Mountain Car · IPC multi-hill Mountain Car instances 2–5 | success probability: 0.00 -> 1.00; 0.00 -> 1.00; 0.00 -> 0.80; 0.00 -> 1.00 | Helped |
| use temporally-extended actions instead of primitive actions | CEM planning on IPC multi-hill Mountain Car · IPC multi-hill Mountain Car instance 1 | reward: 90.98 ± 0.26 -> 91.74 ± 1.55 | No clear effect |
| use temporally-extended actions instead of primitive actions | CEM planning on Dubins Car · u-shaped map | success probability: 1 -> 0.8 | No clear effect |
| add 100 dummy action variables while using temporally-extended actions | A_TE CEM planner · Dubins Car u-shaped map | success probability: 0.8 -> 1.0 | No clear effect |
| use temporally-extended actions rather than primitive actions | MBRL planning · Half Cheetah | training time (hours): 45 -> 5 | Helped |
| use temporally-extended actions rather than primitive actions | MBRL planning · Ant | training time (hours): 41 -> 4 | Helped |
| use bandit selection of the maximum action duration instead of a fixed, selected maximum | A_TE MBRL · Half Cheetah | training time (hours): 5 -> 5.5 | Hurt |
| use a smaller gamma_2 while holding gamma_1 fixed | A_TE planning · Dubins Car cave-mini map | decision steps: 20.2 ± 0.84 -> 23.2 ± 0.45 | Hurt |
| use a smaller gamma_1 while holding gamma_2 fixed | A_TE planning · Dubins Car cave-mini map | primitive steps: 121.8 ± 2.95 -> 115.4 ± 0.55 | Helped |
| increase the planning horizon for A_TE | A_TE CEM planner · Dubins Car u-shaped map | success probability: 0 at D=25; 0.8 at D=50, 75 and 100 | Mixed |
Surprising findings
- Temporally-extended actions reduced failures in Reacher and Pusher offered no gains, with the agent choosing small durations there.
- Non-stationary bandit selection of the maximum duration converges more slowly than a hand-tuned fixed value, though it reaches similar performance.
Limitations
- Temporally-extended actions do not guarantee improvement; trajectories are less flexible and Reacher/Pusher show no gain.
- The bandit formulation requires a fixed planning horizon D_TE per arm; joint selection of duration range and horizon is left to future work.
- Environments with frame-skip set to 1 and capped episode lengths of 1000 deviate from default MuJoCo settings.
Experimental setup
Datasets: Gymnasium Cartpole Gymnasium Mountain Car IPC 2023 multi-hill Mountain Car Dubins Car MuJoCo Ant, Half Cheetah, Hopper, Reacher, Pusher, Walker
Benchmarks: Gymnasium Cartpole Gymnasium Mountain Car IPC 2023 multi-hill Mountain Car Dubins Car u-shaped and cave-mini maps MuJoCo Ant, Half Cheetah, Hopper, Reacher, Pusher, Walker
Metrics: episode reward decision steps computation time per episode success probability
Baselines: A_STD (PETS TS-infinity with primitive actions) Action repeat / frame-skip planning Soft Actor-Critic based timescale control
Builds on: PETS CEM UCB Soft Actor-Critic
Closest papers by meaning
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.89)
- On the role of planning in model-based deep reinforcement learning (ICLR 2021 · similarity 0.89)
- Continuous-time Model-based Reinforcement Learning (ICML 2021 · similarity 0.88)
- A KL-regularization framework for learning to plan with adaptive priors (ICML 2026 · similarity 0.88)
- Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning (ICML 2025 · similarity 0.87)
- When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.