Research map / Reinforcement learning / Visual and representation RL
Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Atari-PB, a unified benchmark that pre-trains a ResNet-50 on 10M Atari transitions with eight objectives across image, video, demonstration and trajectory data, then evaluates generalization to ID, Near-OOD and Far-OOD games.
Problem
Generalization of pre-training objectives for vision-based RL is unclear because prior evaluations stay in-distribution and use non-unified datasets, architectures, and setups.
Approach
Holds dataset, architecture (ResNet-50 with group norm, game-specific neck/head) and downstream protocol fixed while comparing many pre-training objectives, and tests them on three shift levels; finds task-agnostic objectives (image, video) generalize better than task-specific ones (demonstration, trajectory), which help mainly in similar games.
Key results
- Image and video objectives (CURL, MAE, ATC, SiamMAE, R3M) consistently beat the Random frozen baseline across ID, Near-OOD and Far-OOD, with video methods slightly surpassing E2E fine-tuning.
- Demonstration and trajectory objectives give the best ID performance (trajectory methods strongest in ID) but lose advantage in Far-OOD games.
- Offline BC and online RL scores correlate with Pearson 0.85 (ID), 0.61 (Near-OOD) and 0.63 (Far-OOD).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train with image-based objectives CURL or MAE instead of using a randomly initialized frozen backbone | Atari-PB models · ID, Near-OOD, and Far-OOD | IQM normalized score | Helped |
| pre-train with video-based objectives that capture temporal information | Atari-PB models · ID, Near-OOD, and Far-OOD | IQM normalized score | Helped |
| pre-train with demonstration objectives that use actions | Atari-PB models · ID and Near-OOD versus Far-OOD | IQM normalized score | Mixed |
| pre-train with reward-specific trajectory objectives | Atari-PB models · ID versus Near-OOD and Far-OOD | IQM normalized score | Mixed |
| use Expert data instead of Mixed data for pre-training (ablation) | CURL, ATC, BC, and CQL-D · ID | Offline BC performance | Helped |
| use Expert data instead of Mixed data for pre-training (ablation) | CURL, ATC, BC, and CQL-D · Near-OOD and Far-OOD | Offline BC performance | Hurt |
| use a Mixed dataset containing optimal and suboptimal transitions instead of a more uniform Expert dataset (ablation) | CURL, ATC, BC, and CQL-D · Far-OOD | Offline BC performance | Helped |
| increase pre-training dataset size (ablation) | CURL, ATC, BC, and CQL-D · ID | Offline BC performance | Helped |
| increase pre-training dataset size (ablation) | CURL, ATC, BC, and CQL-D · Near-OOD and Far-OOD | Offline BC performance | Mixed |
| scale the model from ResNet-18 to ResNet-50 (ablation) | Atari-PB pre-training methods · ID and Near-OOD | Offline BC performance | Helped |
| scale the model from ResNet-18 to ResNet-50 (ablation) | Atari-PB pre-training methods · Far-OOD | Offline BC performance | No clear effect |
Surprising findings
- The smaller 3-layer CNN often outperforms the ResNet-50 in end-to-end online RL fine-tuning.
- Using expert-only data improves ID performance but hurts Near-OOD and Far-OOD generalization compared with a mixed-optimality dataset.
Limitations
- Pre-training uses a fixed DQN-replay data source and a single architecture family (ResNet-50), limiting generality.
- Far-OOD results do not show clear benefit from larger models, and larger datasets gave mixed OOD effects, which the authors attribute to limited model capacity over 10 epochs.
Experimental setup
Datasets: DQN-Replay-Dataset Atari 2600 games
Benchmarks: Atari-PB (ID, Near-OOD, Far-OOD Atari games)
Metrics: Inter Quantile Mean (IQM) of normalized scores Pearson correlation
Baselines: Random E2E CNN3 CURL MAE ATC SiamMAE R3M BC SPR IDM CQL-M
Builds on: CURL MAE ATC SiamMAE R3M SPR Conservative Q-Learning Decision Transformer
Closest papers by meaning
- Investigating Multi-task Pretraining and Generalization in Reinforcement Learning (ICLR 2023 · similarity 0.93)
- Efficient Reinforcement Learning Through Adaptively Pretrained Visual Encoder (AAAI 2025 · similarity 0.90)
- Pretraining Representations for Data-Efficient Reinforcement Learning (NeurIPS 2021 · similarity 0.90)
- The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining (ICLR 2025 · similarity 0.87)
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.