Research map / Reinforcement learning / Visual and representation RL
Learning Generalizable and Composable Abstractions for Transfer in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A doctoral consortium summary describing research on learning symbolic state and temporal abstractions (options) via CAT+RL and COPlanLearn for sample-efficient transfer and composable RL in factored SSP domains.
Problem
Reinforcement learning agents are sample-inefficient and transfer poorly to new problems, especially in long-horizon sparse-reward settings, because learned abstractions have limited reusability.
Approach
Automatically construct semantically meaningful state abstractions (Conditional Abstraction Tree) and then invent symbolically represented options with abstract policies, recomposing them via bi-directional search over learned option representations while inventing new options as needed.
Key results
- On Taxi World, Mountain Car and other domains, the COPlanLearn/BDPLearn approach solves problems with superior sample efficiency compared to CAT+RL, Option-Critic and DQN across 10 independent trials of 20 sequential problems (2 source, 18 target).
- CAT+RL enables vanilla Q-learning to outperform SOTA methods by drawing out similarities across the state space.
- Learned options in Taxi World exhibit interpretability, composability, and generalizability, transferring from less to more cluttered and smaller to larger environments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn a Conditional Abstraction Tree during reinforcement learning (CAT+RL) | vanilla Q-learning · Stochastic Shortest Path problems | performance | Helped |
| learn and recompose symbolically represented options, using bidirectional search and inventing new options as needed | COPlanLearn · Maze World, Rooms World, Office World, Taxi World, and Mountain Car | sample efficiency | Helped |
Limitations
- Work so far restricted to discrete action spaces.
- Results are preliminary and summarized from prior papers; no full experimental details in this summary.
- Does not yet learn relational options or theoretical guarantees.
Experimental setup
Datasets: Maze World Rooms World Office World Taxi World Mountain Car
Benchmarks: Maze World Rooms World Office World Taxi World Mountain Car
Metrics: fraction of problems solved vs learning episodes sample efficiency
Baselines: CAT+RL Option-Critic DQN
Builds on: Option-Critic CAT+RL Q-learning
Closest papers by meaning
- Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning (AAAI 2025 · similarity 0.90)
- Generalisation in Lifelong Reinforcement Learning through Logical Composition (ICLR 2022 · similarity 0.89)
- State Abstractions for Lifelong Reinforcement Learning (ICML 2018 · similarity 0.88)
- Towards Generalizable Reinforcement Learning via Causality-Guided Self-Adaptive Representations (ICLR 2025 · similarity 0.87)
- Model-Based Transfer Learning for Contextual Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- On Generalization Across Environments In Multi-Objective Reinforcement Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.