Research map / Reinforcement learning / Visual and representation RL

VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: Robust visual encoders for RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VIGOR, a TD-MPC2-based model-based RL method that uses asymmetric weak-to-strong augmentation, dynamics-level latent consistency, and encoder stabilization to achieve zero-shot generalization to unseen visual distractions.

Problem

Model-based RL agents degrade sharply under unseen visual distractions because OOD encoder outputs compound through recursive latent rollouts over the planning horizon.

Approach

Batch-split augmentation pairs weak-only and weak-to-strong latents; a dynamics consistency loss regresses multi-step rollout predictions onto weakly augmented targets, and an encoder alignment loss (SODA-style) prevents encoder drift caused by cross-augmentation supervision.

Key results

• VIGOR reaches mean zero-shot return 777.6 on DMC vs 752.0 for the second-best baseline SRM (+3.4%).
• On Robosuite VIGOR reaches mean 131.8 vs 91.8 for SRM (+43.6%).
• Strong-only augmentation without batch splitting reaches only 48% of VIGOR's DMC performance in ablation.

Surprising findings

• TD-MPC2, the backbone, ranks lowest among all baselines on both DMC and Robosuite zero-shot scores.
• Applying strong augmentation uniformly across the batch (strong-only) severely degrades performance relative to weak-only, indicating naive MFRL augmentation transfer fails in MBRL.

Limitations

• Targets appearance-based perturbations; does not address geometric distortions such as camera rotation.
• Evaluated only in simulation (DMC and Robosuite); no real-robot validation.
• Built only on TD-MPC2; not validated on Dreamer-style backbones.

Experimental setup

Datasets: DeepMind Control Suite Robosuite Places

Benchmarks: DeepMind Control Suite Robosuite RL-ViGen zero-shot protocol

Metrics: mean episode return interquartile mean normalized IQM score

Baselines: DrQ-v2 SVEA SRM SimGRL TD-MPC2 DreamerV3 HRSSM Dr. G

Builds on: TD-MPC2 DrQ-v2 SODA SVEA Dr. G

Closest papers by meaning

• Zero Shot Generalization of Vision-Based RL Without Data Augmentation (ICML 2025 · similarity 0.89)
• The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
• Focus On What Matters: Separated Models For Visual-Based RL Generalization (NeurIPS 2024 · similarity 0.89)
• What Effects the Generalization in Visual Reinforcement Learning: Policy Consistency with Truncated Return Prediction (AAAI 2024 · similarity 0.88)
• Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Image Generation (NeurIPS 2026 · similarity 0.88)
• Policy-shaped prediction: avoiding distractions in model-based reinforcement learning (NeurIPS 2024 · similarity 0.88)