Research map / Reinforcement learning / Visual and representation RL
Handling Delay in Real-Time Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes temporal skip connections plus history-augmented observations so that layer-parallel (pipelined) neural policies in real-time RL suffer less observational delay, evaluated with SAC and PPO on Mujoco, MinAtar and MiniGrid.
Problem
Real-time RL with pipelined/parallel layer computation gives higher throughput but introduces observational delay proportional to network depth, degrading policy performance.
Approach
Use temporal skip connections that pass activations forward in time so the action depends on more recent observations, combined with augmenting observations with recent actions/states, and train the actor by unrolling the parallel computation over sub-trajectories while the critic is trained without delay.
Key results
- Three-layer MLP with proj-from-obs skip connections and augmentation reaches normalized performance of 0.79 ± 0.04 on Mujoco and 0.96 ± 0.002 on MiniGrid (proj-to-action with residual), with MinAtar at 0.52 ± 0.06.
- Parallel computation on GPU gives up to 350% inference speed-up for a 100-layer MLP using sparse weight matrices.
- Vanilla SAC with skip connections reaches 5719 on Ant-v4 vs 3595 for vanilla SAC without skip connections, while on HalfCheetah it is slightly lower (11250 vs 11739).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add temporal skip connections to the default MLP/CNN | Parallel-computation RL agents · Mujoco and MinAtar | normalized return | Helped |
| Add history-augmented observations and/or actions to skip-connected agents | Parallel-computation RL agents · Mujoco and MinAtar | normalized return | Helped |
| Add history augmentation to the default architecture without skip connections (ablation) | Default MLP/CNN · Mujoco and MinAtar | performance | No clear effect |
| Add history augmentation or LSTM to the default architecture (ablation) | Default architecture · MiniGrid | performance | Helped |
| Use projections to action with residual connections instead of projections from observations (ablation) | Temporal skip connections · MinAtar | normalized performance: 0.45 ± 0.05 -> 0.52 ± 0.06 | Helped |
| Use projections to action with residual connections instead of projections from observations (ablation) | Temporal skip connections · MiniGrid | normalized performance: 0.91 ± .006 -> 0.96 ± .002 | Helped |
| Use all forward skip connections between layers rather than projections from observations (ablation) | Temporal skip connections · Mujoco | normalized performance: 0.79± 0.04 -> 0.75 ± 0.05 | Hurt |
| Add traditional skip connections to instantaneous SAC (ablation) | SAC · Ant-v4 | return: 3595 ± 1027 -> 5719 ± 176 | Helped |
| Add traditional skip connections to instantaneous SAC (ablation) | SAC · HalfCheetah-v4, Walker2d-v4, Hopper-v4 | return: 11739 ± 283 -> 11250 ± 32; 4415 ± 227 -> 4597 ± 100; 2672 ± 463 -> 2451 ± 52 | Mixed |
| Increase skip-connected MLP depth from two to three layers (ablation) | Augmented MLP with skip connections, neuron execution time four · Mujoco | performance | Helped |
| Increase skip-connected MLP depth beyond three layers (ablation) | Augmented MLP with skip connections · Mujoco | performance | No clear effect |
| Combine LSTM recurrence with skip connections (ablation) | Skip-connected CNN · MinAtar | performance | Hurt |
Surprising findings
- Traditional skip connections alone do not explain gains: vanilla SAC with skip connections performs similarly to or worse than plain SAC on most Mujoco tasks except Ant.
- Distilling a strong vanilla SAC HalfCheetah policy into the skip-connected agent yields only 7590, comparable to direct SAC training (7892), suggesting an expressivity bottleneck rather than an algorithm issue.
Limitations
- Assumes a fixed neuron execution time; stochastic execution time is not handled.
- Experiments limited to at most five-layer networks.
- Performance gap to instantaneous agents remains notable on MinAtar and HalfCheetah without environment acceleration.
Experimental setup
Datasets: MuJoCo HalfCheetah-v4 MuJoCo Walker2d-v4 MuJoCo Ant-v4 MuJoCo Hopper-v4 MinAtar (six games) MiniGrid Empty-Random-5x5-v0 MiniGrid DoorKey-5x5-v0 Atari subset
Benchmarks: MuJoCo HalfCheetah-v4 MuJoCo Walker2d-v4 MuJoCo Ant-v4 MuJoCo Hopper-v4 MinAtar MiniGrid Atari
Metrics: normalized average return average return inference speed-up
Baselines: SAC PPO RLRD LSTM policy three-layer MLP with augmentation sequential three layers w/ aug
Builds on: Soft Actor-Critic PPO RLRD DAgger
Closest papers by meaning
- Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference (ICLR 2025 · similarity 0.87)
- Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays (ICML 2024 · similarity 0.85)
- Not Only Where, But When: Temporal Scheduling for RLVR (NeurIPS 2026 · similarity 0.85)
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.85)
- Real-Time Recurrent Reinforcement Learning (AAAI 2025 · similarity 0.85)
- SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.