Research map / Reinforcement learning / Visual and representation RL
Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Sliding Puzzles Gym, a visual RL benchmark turning the 3x3 sliding tile puzzle into image-tile observations with adjustable image pool size, and evaluates SAC, PPO, and DreamerV3 variants for representation learning and generalization.
Problem
Existing visual RL benchmarks confound representation learning with policy optimization, environment dynamics, or task difficulty, so there is no way to isolate and scale visual diversity while keeping the task fixed.
Approach
A sliding-puzzle environment whose tiles are image patches drawn from a configurable image pool, so visual diversity can be scaled independently while transition dynamics, action space, and reward stay fixed; the reward is based on a normalized Manhattan distance to the goal.
Key results
- DreamerV3 reaches 80% success at 1.44M steps with pool size 10 versus 2.03M for SAC and 9.73M for PPO, and degrades least as pools grow.
- SAC+RAD (grayscale and channel shuffle augmentation) is the most sample-efficient SAC variant, e.g. 0.82M steps at pool size 10 versus 2.03M for standard SAC.
- Hard OOD success on unseen ImageNet images is near 0% for all PPO, SAC, and DreamerV3 variants; Easy OOD success correlates with sample efficiency (Pearson r=-0.81).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add in-distribution pretrained encoder | PPO · ImageNet-1k validation, pool sizes 1, 5, and 10 | million steps to 80% success: 1.75 0.44 -> 0.95 0.21; 7.80 1.08 -> 5.55 1.22; 9.73 0.36 -> 9.17 1.10 | Helped |
| add RAD data augmentation | SAC · ImageNet-1k validation, pool sizes 1, 5, and 10 | million steps to 80% success: 0.33 0.07 -> 0.24 0.03; 0.91 0.12 -> 0.42 0.06; 2.03 0.38 -> 0.82 0.18 | Helped |
| add CURL contrastive learning | SAC · ImageNet-1k validation, pool sizes 1 and 5 | million steps to 80% success: 0.33 0.07 -> 0.46 0.10; 0.91 0.12 -> 1.56 0.31 | Hurt |
| add SPR self-supervised prediction | SAC · ImageNet-1k validation, pool sizes 1 and 5 | million steps to 80% success: 0.33 0.07 -> 2.09 0.81; 0.91 0.12 -> 3.68 1.68 | Hurt |
| add DBC state metric learning | SAC · ImageNet-1k validation, pool sizes 1 and 5 | million steps to 80% success: 0.33 0.07 -> 0.99 0.25; 0.91 0.12 -> 1.12 0.22 | Hurt |
| add deterministic autoencoder reconstruction | SAC · ImageNet-1k validation, pool sizes 1 and 5 | million steps to 80% success: 0.33 0.07 -> 1.04 0.24; 0.91 0.12 -> 1.02 0.19 | Hurt |
| add variational autoencoder reconstruction | SAC · ImageNet-1k validation, pool sizes 1 and 5 | million steps to 80% success: 0.33 0.07 -> 1.13 0.14; 0.91 0.12 -> 5.30 0.68 | Hurt |
| add out-of-distribution pretrained encoder | PPO · ImageNet-1k validation, pool sizes 1 and 5 | million steps to 80% success: 1.75 0.44 -> 1.34 0.42; 7.80 1.08 -> 7.03 1.07 | Helped |
| remove decoder gradients (ablation) | DreamerV3 · ImageNet-1k validation, pools 1, 5, and 10 | million steps to 80% success: 0.42 0.06 -> 1.13 0.12; 1.23 0.20 -> 1.79 0.61; 1.44 0.58 -> 2.57 0.91 | Hurt |
| increase image pool size | PPO, SAC, and DreamerV3 · ImageNet-1k validation, 3×3 puzzles, pool 1 to pool 10 | million steps to 80% success: PPO: 1.75 0.44 -> 9.73 0.36; SAC: 0.33 0.07 -> 2.03 0.38; DreamerV3: 0.42 0.06 -> 1.44 0.58 | Hurt |
| increase puzzle grid from 3×3 to 4×4 | PPO, SAC, and DreamerV3 · ImageNet-1k validation, pool 1 | million steps to 80% success: PPO: 1.75 0.44 -> 24.46 7.58; SAC: 0.33 0.07 -> 8.14 3.64; DreamerV3: 0.42 0.06 -> 2.26 0.29 | Hurt |
| increase training image pool size | PPO and SAC variants · Easy OOD | success rate: PPO: 0.49 0.13 -> 0.34 0.08; SAC: 0.45 0.12 -> 0.46 0.12 | Hurt |
Surprising findings
- Sophisticated representation learning methods such as CURL, SPR, DBC, and VAE often underperform standard SAC or simple augmentation (RAD).
- Performance on augmented training images often decreases as the training pool grows, and agents trained on smaller pools show more robustness to simple perturbations.
Limitations
- Baselines used minimal hyperparameter tuning, so peak performance of each method may not be reached.
- Only 5 seeds per configuration, with high stochasticity from sampled image pools.
- Hard OOD results are near zero for all methods, so the generalization analysis gives little discrimination between methods.
Experimental setup
Datasets: ImageNet-1k validation split DiffusionDB
Benchmarks: SPGym
Metrics: million environment steps to 80% success rate success rate (Easy OOD, Hard OOD) linear probe accuracy steps per episode
Baselines: PPO SAC RAD CURL SPR DBC SAC-AE SAC-VAE DreamerV3
Builds on: DreamerV3 RAD CURL SAC-AE DBC
Closest papers by meaning
- The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.88)
- DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors (NeurIPS 2024 · similarity 0.88)
- Focus On What Matters: Separated Models For Visual-Based RL Generalization (NeurIPS 2024 · similarity 0.88)
- NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation (NeurIPS 2025 · similarity 0.87)
- When Should We Prefer State-to-Visual DAgger over Visual Reinforcement Learning? (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.