Research map / Reinforcement learning / Visual and representation RL
A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SimGRL, which combines a feature-level frame stack encoder with a shifted random overlay augmentation on top of SVEA to improve zero-shot generalization of model-free visual RL under dynamic background perturbations.
Problem
Visual RL agents generalize poorly to dynamically changing backgrounds, due to imbalanced saliency across stacked frames and observational overfitting to background regions.
Approach
Encode each stacked frame individually in shallow encoder layers before stacking feature maps (feature-level frame stack), and augment each frame with a differently shifted crop of a natural image (shifted random overlay) so that background motion is independent of the task object.
Key results
- On DMControl-GB Video Hard, SimGRL achieves about 15% average gain over SGQN; e.g., Cheetah, Run 301 vs 168 for the next best (CG2A).
- Cartpole, Swingup at Video Hard reaches 727 vs 544 for SGQN.
- SimGRL reaches the same 500K training frames with 44% less wall-clock time than SGQN (9.54 vs 6.16 FPS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add feature-level frame stacking to SVEA (ablation) | SVEA · DMControl-GB Video Hard; six-task results | return: 393±45 -> 719±34 (Cartpole, Swingup); 112±12 -> 264±13 (Cheetah, Run) | Helped |
| Replace random overlay with shifted random overlay (ablation) | SVEA · DMControl-GB Video Hard; six-task results | return: 393±45 -> 708±4 (Cartpole, Swingup); 112±12 -> 269±11 (Cheetah, Run) | Helped |
| Add feature-level frame stacking and shifted random overlay (ablation) | SVEA · DMControl-GB Video Hard; six-task results | return: 393±45 -> 727±23 (Cartpole, Swingup); 112±12 -> 301±7 (Cheetah, Run) | Helped |
| Use irregularly shifted random overlay instead of regular shifted random overlay (ablation) | SimGRL-S · DMControl-GB Video Hard; six-task results | return: 727±36 -> 715±31 (Walker, Walk); 269±11 -> 232±17 (Cheetah, Run) | No clear effect |
| Use arbitrarily stacked random overlay instead of regular shifted random overlay (ablation) | SimGRL-S · DMControl-GB Video Hard; six-task results | return: 727±36 -> 732 ±25 (Walker, Walk); 269±11 -> 237±21 (Cheetah, Run) | No clear effect |
| Apply feature-level frame stacking and shifted random overlay to SGQN (ablation) | SGQN · DMControl-GB Video Hard; six tasks | return: 157±69 -> 311±6 (Cheetah, Run); 544±43 -> 678±25 (Cartpole, Swingup) | Helped |
| Increase the number of image-encoder layers in feature-level frame stacking (ablation) | SimGRL-F · Walker, Walk — Video Hard | generalization gap | Helped |
Surprising findings
- Shifted overlay alone (SimGRL-S) improves generalization despite persisting imbalanced saliency, which the authors attribute to explicit injection of dynamic background elements.
- Image-level frame stacking in DBC appears to cause imbalanced saliency, whereas RNN-based robust RL methods (TIA, DRIBO, RePo) do not show it.
Limitations
- Gains on Video Easy are marginal; performance on Walker, Walk and Cheetah, Run remains unsaturated at Video Hard.
- Generalization under extreme distortions (high-intensity DistractingCS) remains unsolved.
- Robotic manipulation gains are modest and the benchmark mainly features color/texture changes rather than dynamic backgrounds.
Experimental setup
Datasets: DMControl Generalization Benchmark (DMControl-GB) video easy/hard Places (natural image overlay) DistractingCS robotic manipulation tasks (Reach, Peg in Box)
Benchmarks: DMControl-GB Video Easy DMControl-GB Video Hard DistractingCS robotic manipulation (Reach, Peg in Box)
Metrics: average return zero-shot generalization score TID score TID variance generalization gap throughput FPS
Baselines: SAC RAD DrQ SODA SVEA TLDA SGQN EAR CG2A
Builds on: SVEA SAC random overlay DrQ guided backpropagation
Closest papers by meaning
- Focus On What Matters: Separated Models For Visual-Based RL Generalization (NeurIPS 2024 · similarity 0.89)
- DMR: Decomposed Multi-Modality Representations for Frames and Events Fusion in Visual Reinforcement Learning (CVPR 2024 · similarity 0.88)
- Seek Commonality but Preserve Differences: Dissected Dynamics Modeling for Multi-modal Visual RL (NeurIPS 2024 · similarity 0.88)
- The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization (NeurIPS 2024 · similarity 0.88)
- What Effects the Generalization in Visual Reinforcement Learning: Policy Consistency with Truncated Return Prediction (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.