Research map / Reinforcement learning / Visual and representation RL
EvoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EvoControl, a bi-level controller where a slow PPO high-level policy outputs targets for a fast low-level neural policy trained with Evolution Strategies (PGPE), annealed from a PD controller, for high-frequency continuous control.
Problem
High-frequency continuous control suffers from long-horizon credit assignment and poor exploration with end-to-end RL, while fixed PD low-level controllers limit expressiveness and require manual tuning.
Approach
Train a high-level PPO policy and a low-level proprioceptive policy with Evolution Strategies in alternating stages; the low-level policy starts as a PD controller and is gradually annealed (parameter alpha) into a learned neural network, combining PD-driven exploration with ES-based long-horizon optimization of the episodic return.
Key results
- On the same 1M high-level steps, EvoControl (Full State) reaches normalized return 368 on Ant versus 100 for the best fixed-controller baseline, with the best non-EvoControl baseline normalized to 100.
- On Safety Critical Reacher, EvoControl (Target + Proprio.) scores 237 versus 100 for PD Position and 35.9 for 31.25Hz direct torque control.
- Direct torque control at 500Hz scores 0 on Ant and Inverted Pendulum-style tasks in Table 3, indicating poor exploration at high frequency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the fixed low-level PD controller with EvoControl (Full State) | PPO high-level policy · Ant | normalized evaluation return: 100±6.56 -> 368±10.6 | Helped |
| replace the fixed low-level PD controller with EvoControl (Full State) | PPO high-level policy · HalfCheetah | normalized evaluation return: 100±1.21 -> 157±1.1 | Helped |
| replace the fixed low-level PD controller with EvoControl (Full State) | PPO high-level policy · Hopper | normalized evaluation return: 100±1.00 -> 263±1.46 | Helped |
| replace the fixed low-level PD controller with EvoControl (Full State) | PPO high-level policy · Humanoid | normalized evaluation return: 100±2.96 -> 123±2.7 | Helped |
| replace the fixed low-level PD controller with EvoControl (Full State) | PPO high-level policy · Pusher | normalized evaluation return: 100±6.65 -> 262±8.04 | Helped |
| replace the fixed low-level PD controller with EvoControl (Full State) | PPO high-level policy · Safety Critical Reacher | normalized evaluation return: 100±19.2 -> 205±14.1 | Helped |
| use residual-state observations for the low-level policy (ablation) | EvoControl · Reacher 1D | normalized evaluation return: 106±0.936 -> 104±1.19 | No clear effect |
| use target and proprioceptive observations for the low-level policy (ablation) | EvoControl · Reacher 1D | normalized evaluation return: 106±0.936 -> 105±0.776 | No clear effect |
| use target observations for the low-level policy (ablation) | EvoControl · Reacher 1D | normalized evaluation return: 106±0.936 -> 105±0.78 | No clear effect |
| learn PD gains as the low-level policy output (ablation) | EvoControl · Reacher 1D | normalized evaluation return: 106±0.936 -> 105±1.21 | No clear effect |
| use a delta-position controller variant (ablation) | EvoControl · Reacher 1D | normalized evaluation return: 106±0.936 -> 99.1±2.44 | Hurt |
Surprising findings
- Direct torque control at 500Hz performs worse than at 31.25Hz on most environments, showing high frequency alone hurts end-to-end RL.
- PD Position Delta and PD Integrated Velocity fixed controllers collapse to near-zero returns on several tasks, showing sensitivity to controller choice.
Limitations
- Relies on the existence of a fixed PD controller for the task to anneal from.
- Higher computational complexity than PPO alone, and ES is less sample efficient than RL.
Experimental setup
Datasets: MuJoCo/Brax Gym tasks (Ant, HalfCheetah, Hopper, Humanoid, Humanoid Standup, Inverted Double Pendulum, Inverted Pendulum Reacher 1D Safety Critical Reacher Safety Critical HalfCheetah
Benchmarks: Ant HalfCheetah Hopper Humanoid Humanoid Standup Inverted Double Pendulum Inverted Pendulum Pusher Reacher Reacher 1D Walker2D Safety Critical Reacher
Metrics: normalized evaluation return (0-100 scale) state visitation histograms
Baselines: PD Position controller PD Position Delta PD Integrated Velocity PD Position & Kp direct torque control (500Hz) direct torque control (31.25Hz) random policy
Builds on: PPO Evolution Strategies (PGPE) hierarchical reinforcement learning fixed PD controllers
Closest papers by meaning
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.88)
- Reinforcement Learning for Control with Multiple Frequencies (NeurIPS 2020 · similarity 0.87)
- World Models as Reference Trajectories for Rapid Motor Adaptation (NeurIPS 2025 · similarity 0.86)
- Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems (NeurIPS 2024 · similarity 0.85)
- A Differential and Pointwise Control Approach to Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- The Laplacian Keyboard: Beyond the Linear Span (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.