Research map / Reinforcement learning / Visual and representation RL
Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sequence Reinforcement Learning (SRL), which trains a GRU actor to output multi-step action sequences from one state using a learned model and critic to assign per-action credit, then acts model-free at lower decision frequencies, evaluated with a new Frequency-Averaged Score (FAS).
Problem
Standard RL agents require very fast decision timesteps that are impractical on low-compute hardware, and they degrade sharply when decision frequency drops, so control at human-like or slower decision frequencies remains poor.
Approach
An actor-critic in which a recurrent (GRU) actor outputs a J-step action sequence from a single state, while a learned environment model rolls out intermediate states so the critic can assign a learning signal to each primitive action ("temporal recall"); after training the actor runs without the model.
Key results
- SRL-16 achieves FAS of 0.88 on Pendulum and 0.84 on Lunar Lander, versus 0.44 for SAC-1 on Pendulum and 0.18 on Lunar Lander.
- On the Ant environment SRL-16 reaches FAS 0.54 versus 0.16 for the best SAC configuration (SAC-8), and on Humanoid 0.46 versus 0.18 for SAC-16.
- Compared with model-based online planning, SRL has higher FAS on 6 of 10 environments (e.g., Ant 0.54 vs 0.34, Humanoid 0.46 vs 0.18) while online planning is better on Hopper, InvertedPendulum and Swimmer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SRL action-sequence learning instead of SAC | continuous-control policies · Pendulum | FAS: 0.50 $\pm$ 0.03 -> 0.88 $\pm$ 0.02 | Helped |
| Use SRL action-sequence learning instead of SAC | continuous-control policies · Lunar Lander | FAS: 0.56 $\pm$ 0.09 -> 0.84 $\pm$ 0.03 | Helped |
| Use SRL action-sequence learning instead of SAC | continuous-control policies · Hopper | FAS: 0.26 $\pm$ 0.08 -> 0.57 $\pm$ 0.02 | Helped |
| Use SRL action-sequence learning instead of SAC | continuous-control policies · Walker2d | FAS: 0.23 $\pm$ 0.07 -> 0.28 $\pm$ 0.06 | Helped |
| Use SRL action-sequence learning instead of SAC | continuous-control policies · Ant | FAS: 0.16 $\pm$ 0.01 -> 0.54 $\pm$ 0.13 | Helped |
| Use SRL action-sequence learning instead of SAC | continuous-control policies · HalfCheetah | FAS: 0.04 $\pm$ 0.01 -> 0.28 $\pm$ 0.01 | Helped |
| Use SRL action-sequence learning instead of SAC | continuous-control policies · Humanoid | FAS: 0.18 $\pm$ 0.04 -> 0.46 $\pm$ 0.04 | Helped |
| Use SRL action-sequence learning instead of SAC | continuous-control policies · InvertedPendulum | FAS: 0.34 $\pm$ 0.20 -> 0.44 $\pm$ 0.04 | Helped |
| Use SRL action-sequence learning instead of SAC | continuous-control policies · InvertedDPendulum | FAS: 0.09 $\pm$ 0.01 -> 0.13 $\pm$ 0.02 | Helped |
| Use SRL action-sequence learning instead of SAC | continuous-control policies · Reacher | FAS: 0.87 $\pm$ 0.02 -> 0.96 $\pm$ 0.00 | Helped |
| Use SRL action-sequence learning instead of SAC | continuous-control policies · Swimmer | FAS: 0.54 $\pm$ 0.06 -> 0.42 $\pm$ 0.15 | Hurt |
| Increase action sequence length from 2 to 16 (ablation) | SRL · most continuous-control tasks | FAS | Mixed |
Surprising findings
- SAC-1 is the best configuration on most environments at standard frequency, but its FAS collapses at longer action sequence lengths, and training SAC with larger J trades off short-horizon performance.
- The Gym-style GPM baseline achieved FAS of only about 0.04 on three of four tested environments, below SAC-J.
Limitations
- SRL underperforms on Walker2d and InvertedDoublePendulum at longer sequence lengths, attributed to model error.
- SAC with action repetition outperforms SRL on Swimmer, since SRL does not use action repetition for exploration.
- Training requires longer wall-clock time than SAC due to the model rollouts and extra actor updates.
Experimental setup
Datasets: MuJoCo (OpenAI Gym) continuous control tasks
Benchmarks: MuJoCo Pendulum LunarLanderContinuous Hopper Walker2d Ant HalfCheetah Humanoid InvertedPendulum InvertedDoublePendulum Reacher Swimmer
Metrics: Frequency-Averaged Score (FAS) episode return
Baselines: SAC SAC-J (action repetition) Generative Planning Method (GPM) Model-based online planning TempoRL SAC-R (randomized frame-skipping)
Builds on: Soft Actor-Critic TempoRL Generative Planning Method TCRL
Closest papers by meaning
- PWM: Policy Learning with Multi-Task World Models (ICLR 2025 · similarity 0.90)
- TD-MPC2: Scalable, Robust World Models for Continuous Control (ICLR 2024 · similarity 0.89)
- Continual Reinforcement Learning by Planning with Online World Models (ICML 2025 · similarity 0.89)
- Improving planning and MBRL with temporally-extended actions (NeurIPS 2025 · similarity 0.89)
- Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network (ICML 2025 · similarity 0.89)
- Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.