Research map / Reinforcement learning / MDPs and planning
When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TaCoS, a framework that reformulates continuous-time SDE control with time-adaptive interaction as an extended discrete-time MDP solvable by standard RL, plus a no-regret optimistic model-based algorithm (OTaCoS).
Problem
Continuous-time systems are typically controlled with a fixed discretization frequency, but each interaction (measurement or action switch) is costly, so the time-adaptive choice of when to sense and control is needed.
Approach
Extend the policy to also predict the duration each action is applied, with time-to-go (and integrated reward, interaction count) in the augmented state, yielding an equivalent discrete-time MDP solvable by any RL algorithm; additionally an optimistic model-based planner with regret bounds for SDE dynamics.
Key results
- On the Pendulum with a constant switch cost of 0.1, the approach reduces interactions from 200 to 24 while still solving the swing-up task.
- Time-adaptive SAC outperforms equidistant discretization under bounded interactions, especially for small K, in greenhouse and pendulum swing-down tasks.
- The model-based OTaCoS is reported to be more sample efficient than PETS-TaCoS and Mean-TaCoS on Pendulum and RC car, and the paper proves sublinear regret under smoothness assumptions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TaCoS to choose interaction times adaptively instead of equidistant discretization | SAC-TaCoS versus equidistant SAC · Greenhouse and Pendulum swing-down, bounded interactions | episode reward | Helped |
| Increase the interaction cost | TaCoS · Greenhouse and Pendulum swing-up | number of interactions | Helped |
| Increase the interaction cost | TaCoS · Greenhouse and Pendulum swing-up | episode reward | Hurt |
| Increase environment stochasticity | TaCoS · Greenhouse and Pendulum swing-up | number of interactions | Hurt |
| Use TaCoS with SAC or PPO across different minimum action durations | SAC-TaCoS and PPO-TaCoS · Brax benchmark environments, varying t_min | performance robustness | Helped |
| Decrease the interaction duration (increase frequency) | Standard RL methods · Brax benchmark environments | performance | Hurt |
| Use TaCoS instead of standard SAC | SAC-TaCoS versus SAC-MC · Brax benchmark environments, learning curve at a specific frequency | reward per physical environment time | Helped |
| Use TaCoS instead of its model-free counterpart | Optimistic TaCoS · Pendulum and RC car, interaction-cost setting | sample efficiency | Helped |
| Use TaCoS instead of standard discrete-time RL | TaCoS · Continuous-time RL experiments, interaction-cost setting | number of interactions | Helped |
Limitations
- Theoretical regret bounds rely on Lipschitz dynamics and well-calibrated GP-type models.
- Model-based experiments limited to Pendulum and RC car.
Experimental setup
Datasets: Pendulum Reacher Halfcheetah Humanoid Greenhouse RC car
Benchmarks: Pendulum Greenhouse Reacher Halfcheetah Humanoid RC car
Metrics: episode reward number of interactions regret
Baselines: SAC PPO SAC-MC PETS-TaCoS Mean-TaCoS
Builds on: SAC PPO PETS
Closest papers by meaning
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.88)
- Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems (NeurIPS 2024 · similarity 0.86)
- DARLING: Detection Augmented Reinforcement Learning with Non-Stationary Guarantees (NeurIPS 2026 · similarity 0.86)
- Time-Constrained Robust MDPs (NeurIPS 2024 · similarity 0.86)
- Improving planning and MBRL with temporally-extended actions (NeurIPS 2025 · similarity 0.86)
- Regret-Free Reinforcement Learning for Temporal Logic Specifications (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.