Research map / Reinforcement learning / Visual and representation RL
The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks frozen pre-trained visual representations (CLIP, R3M, VC-1, DINOv2, etc.) inside DreamerV3 and TD-MPC2 on DMC, ManiSkill2 and Miniworld, finding they are not more sample-efficient or OOD-robust than representations learned from scratch.
Problem
Whether pre-trained visual representations (PVRs), which help model-free RL, improve sample efficiency and out-of-distribution generalization of model-based RL agents.
Approach
Plugs a set of frozen PVRs into the encoder of two MBRL algorithms (DreamerV3, TD-MPC2) via a jointly trained linear layer, evaluates ID and OOD visual randomizations, and analyzes world-model dynamics and reward prediction errors to explain the gap.
Key results
- Representations learned from scratch are equally or more data-efficient than PVRs in most cases, including in-distribution autoencoders, especially on DMC.
- No PVR performs consistently well in both DMC and ManiSkill2/Miniworld OOD settings; VC-1 is the only one that does reasonably well in both, and from-scratch agents outperform PVR agents even OOD.
- Reward prediction error correlates with task performance (r=-0.66, p=0.004), while dynamics prediction error does not (r=-0.22, p=0.4); PVRs produce comparably accurate dynamics but worse reward predictions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use pre-trained visual representations instead of learning the representation from scratch | DreamerV3 and TD-MPC2 · DMC, ManiSkill2, and Miniworld; ID data efficiency | normalized return | Hurt |
| use pre-trained visual representations instead of learning the representation from scratch | DreamerV3 and TD-MPC2 · DMC, ManiSkill2, and Miniworld; OOD evaluation | normalized return | Hurt |
| use VC-1 rather than a representation learned from scratch | DreamerV3 and TD-MPC2 · DMC, ManiSkill2, and Miniworld; OOD evaluation | normalized return | Mixed |
| use a diverse-data PVR rather than a non-diverse-data PVR | PVR-based MBRL agents · DMC and ManiSkill2; OOD evaluation | IQM return | Helped |
| use a ViT-based representation rather than a non-ViT representation | PVR-based MBRL agents · DMC and ManiSkill2; OOD evaluation | IQM return | Helped |
| train the representation on sequential data | PVR-based MBRL agents · ManiSkill2 and Miniworld; OOD evaluation | IQM return | Helped |
| train the representation on sequential data | PVR-based MBRL agents · DMC; OOD evaluation | IQM return | No clear effect |
| use language-conditioned pre-training | PVR-based MBRL agents · DMC, ManiSkill2, and Miniworld; OOD evaluation | IQM return | No clear effect |
| replace the linear encoder layer with an MLP (ablation) | DreamerV3 with VC-1 or a representation learned from scratch · DMC and ManiSkill2; ID and OOD evaluation | normalized return | No clear effect |
| use two-thirds of VC-1 transformer blocks instead of the full model (ablation) | VC-1-based MBRL agent · control tasks | performance | No clear effect |
| use one-third of VC-1 transformer blocks instead of the full model (ablation) | VC-1-based MBRL agent · control tasks | performance | Hurt |
| use representations learned from scratch rather than PVRs | DreamerV3 and TD-MPC2 world models · DMC Pendulum Swingup; reward prediction | reward prediction error | Helped |
Surprising findings
- Language-conditioned PVRs do not generalize better OOD, and ViT-based CLIP performs on par with ResNet-based CLIP, suggesting data diversity matters more than architecture.
- Autoencoders pre-trained in-distribution on task data still underperform training the encoder from scratch in MBRL.
Limitations
- Only three domains (DMC, ManiSkill2, Miniworld) and two MBRL algorithms; no real-world experiments.
- Computational cost restricted the number of PVRs and TD-MPC2 was run with a smaller PVR set on DMC.
- Only visual distribution shifts were studied; the linear encoder choice may not reflect other PVR usages.
Experimental setup
Datasets: DMControl Generalization Benchmark ManiSkill2 Miniworld PickupObjects-v0 Ego4D ImageNet WIT LAION-2B Taskonomy LVD-142M
Benchmarks: DeepMind Control Suite ManiSkill2 Miniworld
Metrics: normalized return (IQM) sample efficiency OOD normalized return dynamics KL/MSE prediction error reward prediction error
Baselines: Encoder from scratch CLIP R3M Taskonomy VIP DINOv2 OpenCLIP VC-1 R2D2
Builds on: DreamerV3 TD-MPC2 R3M VC-1 DMControl Generalization Benchmark
Closest papers by meaning
- Focus On What Matters: Separated Models For Visual-Based RL Generalization (NeurIPS 2024 · similarity 0.90)
- Efficient Reinforcement Learning Through Adaptively Pretrained Visual Encoder (AAAI 2025 · similarity 0.90)
- Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning (ICML 2025 · similarity 0.89)
- What Can RL Bring to VLA Generalization? An Empirical Study (NeurIPS 2025 · similarity 0.89)
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (ICML 2025 · similarity 0.89)
- VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.