Research map / Reinforcement learning / Policy gradient and actor-critic
Coprocessor Actor Critic: A Model-Based Reinforcement Learning Approach For Adaptive Brain Stimulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Coprocessor Actor Critic, a model-based RL method that learns a Q-function for world actions in a biomechanical simulator, then learns a brain stimulation-to-action model online, guided by and recalibrating that Q-function, for adaptive brain stimulation in simulated stroke.
Problem
Adaptive brain stimulation policies for stroke patients must be learned with very few costly patient interactions, while model-free RL needs many environment samples.
Approach
Decouple coprocessor policy learning into learning optimal world actions via a biomechanical simulator (Q-function), then learning a stimulation-to-world-action brain model online, sampling stimulations that maximize the simulator Q-function and iteratively updating the Q-function over the learned brain model's realizable actions.
Key results
- CopAC reaches effective policies in under 25 online episodes across environments, and often matches SAC performance with about half the interactions, with more than 10x sample efficiency in many environments.
- CopAC achieves higher training reward than SAC and MBPO across domains, with the largest gap in the 39-dimensional MyoSim Hand Pose task.
- Ablations show that updating the world-action Q-function matters most in Swimmer, and Q-max sampling outperforms random sampling in several domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CopAC instead of SAC | Adaptive brain stimulation control tasks · Control environments | sample efficiency: 10x | Helped |
| Use CopAC instead of SAC | Adaptive brain stimulation control tasks · Control environments | evaluation reward | Helped |
| Use CopAC instead of MBPO | Adaptive brain stimulation control tasks · MyoSim Pose | evaluation reward | Helped |
| Use CopAC instead of MBPO | Adaptive brain stimulation control tasks · MyoSim Reach and Pendulum | evaluation reward | No clear effect |
| Add online Q-function updates (ablation) | CopAC · Swimmer | evaluation reward | Helped |
| Add online Q-function updates (ablation) | CopAC · Reacher | evaluation reward | Helped |
| Add online Q-function updates (ablation) | CopAC · Several other control domains | evaluation reward | Helped |
| Use Q-maximizing stimulation sampling instead of random sampling (ablation) | CopAC without Q updates · MyoSim Pose | evaluation reward | Helped |
| Use Q-maximizing stimulation sampling instead of random sampling (ablation) | CopAC without Q updates · Swimmer, Reacher, Pendulum, and Inverted Double Pendulum | evaluation reward | Helped |
| Use Q-maximizing stimulation sampling instead of random sampling (ablation) | CopAC · All evaluated environments | training reward | Helped |
| Learn the world-action Q-function using offline RL instead of simulation | CopAC · Most environments | training reward | Helped |
| Learn the world-action Q-function using offline RL instead of simulation | CopAC · Evaluated environments | training reward | Hurt |
Surprising findings
- Random sampling underperforms in several domains, but the benefit of updating the world-action value model is domain-dependent and may be inconsequential when all world actions are realizable.
Limitations
- Evaluated only in silico; no in vivo validation.
- Relies on a high-fidelity biomechanical simulator to learn the world-action Q-function; safety during online learning is not guaranteed.
Experimental setup
Datasets: MyoSuite simulated reaching and pose tasks OpenAI Gym continuous control tasks
Benchmarks: MyoSim Arm Reach MyoSim Hand Pose OpenAI Gym Swimmer OpenAI Gym Reacher OpenAI Gym Pendulum OpenAI Gym Inverted Double Pendulum LunarLander
Metrics: evaluation reward training reward sample efficiency
Baselines: Soft Actor-Critic Model-Based Policy Optimization CopAC without Q update CopAC without Q update and Q max inverse brain model coprocessor
Builds on: Soft Actor-Critic Model-Based Policy Optimization MyoSuite Conservative Soft Actor Critic
Closest papers by meaning
- OMiSO: Adaptive optimization of state-dependent brain stimulation to shape neural population states (NeurIPS 2025 · similarity 0.84)
- Real-Time Recurrent Reinforcement Learning (AAAI 2025 · similarity 0.83)
- Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation (ICML 2024 · similarity 0.83)
- World Models as Reference Trajectories for Rapid Motor Adaptation (NeurIPS 2025 · similarity 0.83)
- Feedback control guides credit assignment in recurrent neural networks (NeurIPS 2024 · similarity 0.83)
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.