Research map / Bandits and online learning / Online convex optimization and games
Online Control with Adversarial Disturbance for Continuous-time Linear Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-level online non-stochastic control algorithm for continuous-time linear systems (DAC policy plus OCO with memory, updated every m steps) achieving sublinear regret, and applies a frame stack-and-skip variant to SAC in MuJoCo domain-randomization tasks.
Problem
Online control of continuous-time linear systems under adversarial (non-stochastic) disturbances, with sublinear regret against the best linear controller in hindsight, plus training RL agents in domain randomization environments.
Approach
Discretize time with step h and combine a Disturbance-Action policy (using disturbance estimates over [t,t+h]) with OCO with memory whose parameters are updated only once every m = Θ(1/h) steps, so the lookahead and update frequency balance discretization, approximation, and OCO regret; empirically, stacking past observations with frame skips is added to SAC.
Key results
- Theory: regret O(sqrt(T) log T) for continuous-time linear control with adversarial disturbances when h = Θ(1/√T) and H = Θ(log T).
- In hopper DR experiments, frame stack increase from 1 to 3 significantly improved performance, and frame skip of 3 was optimal among tested values.
- The stack-and-skip SAC variant (h=3, m=3) outperformed standard SAC across the test environments of hopper, half-cheetah, and walker2d (per the paper's claims).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase frame stack number from 1 to 3 (ablation) | SAC with Stack & skip · Hopper domain randomization | performance | Helped |
| Increase frame stack number beyond 3 (ablation) | SAC with Stack & skip · Hopper domain randomization | performance | No clear effect |
| Set frame skip to 3 rather than too small or too large values (ablation) | SAC with Stack & skip · Hopper domain randomization | performance | Helped |
| Use Stack & skip instead of standard SAC | SAC · Domain-randomized Hopper, Half-Cheetah, and Walker2D test environments | performance | Helped |
Limitations
- Theory assumes known system dynamics and convex costs.
- Empirical evaluation limited to SAC on three MuJoCo tasks with 3 seeds.
- Lacks analysis of unknown dynamics and strongly convex costs (stated as future work).
Experimental setup
Datasets: MuJoCo Hopper MuJoCo Half-Cheetah MuJoCo Walker2D
Benchmarks: MuJoCo Hopper MuJoCo Half-Cheetah MuJoCo Walker2D
Metrics: episodic reward regret (theoretical)
Baselines: Soft Actor-Critic
Builds on: Soft Actor-Critic Disturbance-Action Controller (DAC) OCO with memory strongly stable policy class online non-stochastic control
Closest papers by meaning
- Online Nonstochastic Model-Free Reinforcement Learning (NeurIPS 2023 · similarity 0.90)
- Online Learning in Stabilized Linear Dynamical Games with Adversarial Disturbances (NeurIPS 2026 · similarity 0.90)
- Best of Both Worlds Guarantees for Smoothed Online Quadratic Optimization (ICML 2024 · similarity 0.86)
- Finite Sample Analyses for Continuous-time Linear Systems: System Identification and Online Control (NeurIPS 2025 · similarity 0.86)
- Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning (ICLR 2025 · similarity 0.86)
- Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.