Research map / Reinforcement learning / Visual and representation RL

Visual Reinforcement Learning with Residual Action

AAAI 2025 · Reinforcement learning › Visual and representation RL · approach: Robust visual encoders for RL · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ResAct, which makes a vision-based RL policy output a residual action added to the previous action, with an observation-difference encoder (ODL) that feeds frame-difference features to the policy, improving DeepRAD and other baselines on DMControl and CARLA.

Problem

Learning continuous control policies from high-dimensional visual observations is sample-inefficient because the policy must map images directly to actions whose distribution is task-dependent and unknown a priori.

Approach

Redefine the policy output to be a residual action relative to the previous action (Residual Action Learning), and encode the difference between adjacent frames with two independently parameterized CNNs (Observation Difference Learning) to provide a compact representation for predicting that residual, without constraining the vanilla RL objective.

Key results

• On 6 DMControl tasks at 100K steps, ResAct on DeepRAD averages 822.1 versus 767.8 for DeepRAD and 739.0 for SVEA among the compared methods' best.
• On CARLA highway driving, ResAct reaches episode reward 283 versus 198 for DeepRAD and reduces average brake from 1.03% to 0.30%.
• On 5 DMControl hard tasks at 1M steps, ResAct averages 630.2 versus 566.0 for DeepRAD and 584.8 for TACO.

Measured effects reported in the paper

Surprising findings

• Residual actions follow a near-zero-centered normal distribution while raw actions are spread across the value range, which motivates the approach.

Limitations

• Relies on the assumption that residual actions are small in most cases; in the worst case of dramatic action changes each step, the benefit vanishes to vanilla RL.
• Evaluated only on DMControl and CARLA; applicability to model-based frameworks is left to future work.

Experimental setup

Datasets: DeepMind Control Suite CARLA

Benchmarks: DeepMind Control Suite CARLA

Metrics: episode reward average score standard deviation driving distance crash intensity average steer average brake

Baselines: pixel SAC RAD DeepMDP DeepRAD CURL SVEA PlayVirtual MLR PSRL TACO MaDi Flare

Builds on: Soft Actor-Critic RAD DeepMDP CURL Flare

Closest papers by meaning

• Return-Critic: Bridging Goal Discrepancy for Efficient Visual Reinforcement Learning (ICML 2026 · similarity 0.86)
• Visual Reinforcement Learning with Imagined Goals (NeurIPS 2018 · similarity 0.85)
• Learning from Visual Observation via Offline Pretrained State-to-Go Transformer (NeurIPS 2023 · similarity 0.84)
• When Should We Prefer State-to-Visual DAgger over Visual Reinforcement Learning? (AAAI 2025 · similarity 0.84)
• DMR: Decomposed Multi-Modality Representations for Frames and Events Fusion in Visual Reinforcement Learning (CVPR 2024 · similarity 0.84)
• DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.83)