Research map / Reinforcement learning / Visual and representation RL
Focus On What Matters: Separated Models For Visual-Based RL Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SMG, a separated-model reconstruction approach (foreground/background encoders with mask) plus foreground and Q-value consistency losses on SAC to improve zero-shot visual RL generalization, especially under video-background shifts.
Problem
Visual-based RL agents overfit to task-irrelevant features (colors, textures, backgrounds), so generalization to unseen environments is poor; image reconstruction is avoided for this reason.
Approach
Use two model branches (foreground and background encoders/decoders with a predicted mask) so reconstruction benefits representation learning without absorbing task-irrelevant features, and enforce attribution and Q-value consistency between raw and augmented observations via attribution augmentation.
Key results
- On DMC video-hard, SMG outperforms all baselines on all five tasks, e.g. finger-spin 910 vs 710 for the second-best (SGQN) and cheetah-run 303 vs 135.
- On peg-in-box, SMG averages 234 across six settings versus 116 for the second-best (SRM), a 102% improvement.
- On reach, SMG averages 29 with std 2 versus 21 for SGQN, a 38% improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SMG instead of the strongest baseline | DMControl video-hard · Five DMControl tasks | episode return: +165; +200; +85; +180; +168 | Helped |
| use SMG instead of the strongest baseline | DMControl video-easy · Five DMControl tasks | episode return: +121; +92; +12; +74; +40 | Helped |
| use SMG instead of the strongest baseline | SMG · Peg-in-box average across test settings | episode return: +118 | Helped |
| use SMG instead of the strongest baseline | SMG · Reach average across test settings | episode return: +8 | Helped |
| add foreground consistency loss (ablation) | SMG · DMControl video-hard | episode return: -15% | Helped |
| add empowerment loss (ablation) | SMG · DMControl video-hard | episode return: -22% | Helped |
| add background reconstruction loss (ablation) | SMG · DMControl video-hard | episode return: -23% | Helped |
| add mask ratio loss (ablation) | SMG · DMControl video-hard | episode return: -30% | Helped |
| add Q-value consistency loss (ablation) | SMG · DMControl video-hard | episode return: -56% | Helped |
| vary mask ratio rho from 0.04 to 0.08 (ablation) | SMG · DMControl walker-walk video-hard | episode return | No clear effect |
| set mask ratio rho to 0.02 or 0.1 instead of 0.06 (ablation) | SMG · DMControl walker-walk video-hard | episode return: around 6% | Hurt |
| use SMG instead of the strongest baseline | SMG · DMControl color-easy and color-hard | episode return: 7 out of 10 tasks | Mixed |
Surprising findings
- Ablation shows removing the Q-value consistency loss causes the largest drop (about 56% on average in video-hard), larger than removing the mask or background losses.
- Baselines in robotic manipulation mostly fail zero-shot; only SMG and SGQN achieve generalization in the evaluation settings.
Limitations
- Performance declines when observations contain many task-relevant objects, since learning an accurate mask becomes harder.
- Not evaluated on non-static camera viewpoints or real-world deployment.
- Roughly double parameter count relative to baselines.
Experimental setup
Datasets: DMControl DMControl Generalization Benchmark peg-in-box robotic manipulation reach robotic manipulation
Benchmarks: DMControl Generalization Benchmark peg-in-box reach
Metrics: episode return mean and standard deviation over four seeds
Baselines: SAC DrQ SODA SVEA SRM SGQN
Builds on: SAC DrQ SODA SVEA SGQN
Closest papers by meaning
- The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- What Effects the Generalization in Visual Reinforcement Learning: Policy Consistency with Truncated Return Prediction (AAAI 2024 · similarity 0.90)
- A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations (NeurIPS 2024 · similarity 0.89)
- VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (ICML 2025 · similarity 0.88)
- Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.