Research map / Reinforcement learning / Visual and representation RL
Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Skipper, a model-based RL framework that builds a decision-time proxy graph of generated checkpoints with learned spatially-abstract edge estimates, improving zero-shot generalization on gridworld navigation over LEAP and Director.
Problem
Generalization of RL agents to novel task variations (unseen layouts/difficulty) after training on few tasks; existing hierarchical planners generalize poorly.
Approach
Decompose tasks at decision time via a proxy problem: a graph whose vertices are generated checkpoints (sparse decision points) and whose edges are learned discount/reward estimates over attention-selected partial state features (spatial abstraction), with the proxy graph learned end-to-end from hindsight and pruned with k-medoids.
Key results
- Skipper agents show statistically significant advantage in OOD zero-shot success rate over modelfree, LEAP and Director throughout training on 50 training tasks, with the regen variant performing best.
- Director's generalization deteriorates quickly as the number of training tasks increases, while Skipper improves with more training tasks.
- The modelfree baseline matches Skipper only when trained on a different task in every episode, which is infeasible in practice.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Skipper's spatio-temporal proxy-problem planning | Skipper · 12×12 MiniGrid-BabyAI OOD navigation | success rate | Helped |
| Regenerate the proxy problem when replanning is triggered (ablation) | Skipper · MiniGrid-BabyAI OOD navigation | success rate | Helped |
| Use spatial abstraction via the local perception field (ablation) | Skipper · Generalization across training tasks | generalization performance | Helped |
| Add delusion suppression to estimator training (ablation) | Skipper · MiniGrid-BabyAI OOD navigation | generalization performance | Helped |
| Use k-medoids pruning to select proxy-problem checkpoints (ablation) | Skipper · MiniGrid-BabyAI navigation | performance | Helped |
| Combine LEAP with the ideas proposed in this paper | LEAP · MiniGrid-BabyAI OOD navigation | generalization performance | Helped |
| Train the model-free baseline on a different task each episode | Model-free baseline · MiniGrid-BabyAI OOD navigation | generalization performance | Helped |
| Increase the number of training tasks | Skipper and model-free baseline · MiniGrid-BabyAI OOD navigation | generalization performance | Helped |
Surprising findings
- Skipper behaves suboptimally on training tasks yet generalizes strongly to OOD evaluation tasks.
- Skipper and LEAP perform poorly when trained on only one task, since the context/partial-description split cannot be learned.
Limitations
- Checkpoints are generated at random by sampling the partial description space rather than prioritizing important states.
- Current implementation is limited to pixel-input, fully observable, discrete state and action tasks with sparse rewards.
- The performance bound's practical scope is unclear beyond the sparse-reward navigation setting.
Experimental setup
Datasets: MiniGrid-BabyAI generated gridworld navigation tasks
Benchmarks: MiniGrid-BabyAI lava gridworld 12x12 OOD difficulties 0.25/0.35/0.45/0.55
Metrics: success rate zero-shot generalization success rate
Baselines: modelfree (prioritized distributional Double DQN) Director LEAP
Builds on: Hindsight Experience Replay C51 LEAP Director Double DQN
Closest papers by meaning
- Reconciling Spatial and Temporal Abstractions for Goal Representation (ICLR 2024 · similarity 0.87)
- Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning (AAAI 2025 · similarity 0.87)
- Horizon Generalization in Reinforcement Learning (ICLR 2025 · similarity 0.86)
- PcLast: Discovering Plannable Continuous Latent States (ICML 2024 · similarity 0.86)
- Learning Generalizable and Composable Abstractions for Transfer in Reinforcement Learning (AAAI 2024 · similarity 0.86)
- Reward-Respecting Subtasks for Model-Based Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.