Research map / Reinforcement learning / Visual and representation RL
Zero Shot Generalization of Vision-Based RL Without Data Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALDA, which combines a quantized disentangled autoencoder (QLAE) with a softmax-based Hopfield-style associative memory on top of SAC to achieve zero-shot generalization in vision-based RL without data augmentation.
Problem
Zero-shot generalization of vision-based RL agents to novel visual distribution shifts (color and distractor video/camera shake) without relying on costly data augmentation or large datasets.
Approach
Learn a disentangled, scalar-codebook latent space jointly with the RL policy and add a softmax (modern Hopfield) association step so that OOD latent components are mapped back to in-distribution codebook values; the paper also formally argues data augmentation is a form of weak disentanglement.
Key results
- ALDA outperforms DARLA, SAC+AE, and RePo on both color hard and distracting cs evaluation environments, excluding SVEA.
- ALDA performs comparably to SVEA (trained with Places image overlay augmentation) and in some cases equals it, despite using no augmentation beyond the original task images.
- ALDA with SVEA-style color easy augmentation is shown to be worse than ALDA on training and color hard, and equal on distracting cs (appendix).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ALDA instead of the reported non-SVEA baselines | Vision-based RL on DMControl · Color hard and DistractingCS | return | Helped |
| Use ALDA instead of SVEA with image overlays | Vision-based RL on DMControl · Color hard and DistractingCS | return | Mixed |
| Replace QLAE's argmin retrieval with softmax latent retrieval and omit the quantization loss (ablation) | SAC+QLAE · Color hard | return | Helped |
| Replace QLAE's argmin retrieval with softmax latent retrieval and omit the quantization loss (ablation) | SAC+QLAE · DistractingCS | return | Helped |
| Use QLAE instead of BioAE's non-associative latent dynamics (ablation) | SAC with a disentanglement model · Walker walk, Color hard and DistractingCS | return | Helped |
| Encode stacked frames jointly in ALDA and remove the downstream 1D CNN (ablation) | ALDA · Walker, Color hard and DistractingCS | return | Hurt |
| Use lower latent-model temperatures (β=1, 10, or 50) instead of β=100 (ablation) | ALDA · Walker, Color hard and DistractingCS | return | No clear effect |
| Increase latent dimensionality above z_d=12 (ablation) | ALDA · Walker Walk training and Color Hard | return | Hurt |
| Backpropagate critic gradients through the latent model to the encoder (ablation) | ALDA · Walker Walk training and evaluation | return | Hurt |
| Train SVEA on color easy instead of using image-overlay augmentation (ablation) | SVEA · Training and Color Hard | return | Hurt |
| Train SVEA on color easy instead of using image-overlay augmentation (ablation) | SVEA · DistractingCS | return | No clear effect |
Surprising findings
- Lower softmax temperature beta (1 to 50) has little to no effect on generalization and even increases training performance, challenging assumptions about quantization-based disentanglement.
- Performance degrades severely for all algorithms on the DistractingCS environment, suggesting camera shaking may require finetuning.
Limitations
- Disentangled latent z_d does not explicitly model temporal information; temporal info is handled by a downstream 1D-CNN.
- Latent dimensionality |z_d|=12 is chosen empirically with no rigorous method; no quantitative disentanglement metrics available on DMControl.
- Does not exploit more recent learnable attention-based or energy-based Hopfield networks.
Experimental setup
Datasets: DeepMind Control Suite DMControl Generalization Benchmark Distracting Control Suite DAVIS 2017 Places
Benchmarks: DMControl Generalization Benchmark color hard Distracting Control Suite
Metrics: episode return 95% confidence interval over seeds
Baselines: DARLA SAC+AE RePo SVEA QLAE BioAE
Builds on: Soft Actor-Critic QLAE VQ-VAE DARLA SAC+AE
Closest papers by meaning
- Zero-Shot Reinforcement Learning from Low Quality Data (NeurIPS 2024 · similarity 0.89)
- VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- One Subgoal at a Time: Zero-Shot Generalization to Arbitrary Linear Temporal Logic Requirements in Multi-Task Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.88)
- Provable Zero-Shot Generalization in Offline Reinforcement Learning (ICML 2025 · similarity 0.87)
- Focus On What Matters: Separated Models For Visual-Based RL Generalization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.