Research map / Reinforcement learning / Visual and representation RL
Identifying Selections for Unsupervised Subtask Discovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that subtask/subgoal variables act as selection variables (not confounders or mediators) and proposes seq-NMF to learn them, then uses the learned subgoals for GAIL-based multi-task imitation on the D4RL Kitchen environment.
Problem
Subtask discovery from unlabeled expert demonstrations is typically driven by heuristics or likelihood models that ignore the true data-generating structure, which can mislead segmentation and hurt downstream imitation on new tasks.
Approach
Models subgoals as selection variables (s_t -> g_t <- a_t) identified via conditional independence tests, defines subtasks as minimal sets of options consistent with policy uncertainty, and learns them with a sequential NMF using binary, sparsity, and similarity regularizers; learned subgoals condition a GAIL/PPO imitation policy.
Key results
- On new Kitchen tasks with 4 subtasks, the selection-based subtask method outperforms H-AIRL, Directed-info GAIL and Option-GAIL in episodic return across 5 runs.
- CI tests on Driving data give p-values of 0.00 for s_t ⊥ a_t | g_t and 0.40-0.65 for s_{t+1} ⊥ g_t | s_t, a_t, supporting the selection structure.
- seq-NMF automatically partitions the Driving trajectories into three subtasks at the crossing points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use selection-based subtasks learned from demonstrations to condition the imitation policy | Kitchen multi-task imitation learning · Kitchen new tasks with 4 subtasks | episodic accumulated return | Helped |
| Use selection-based subtasks learned from demonstrations to condition the imitation policy | Kitchen multi-task imitation learning · Kitchen new task with 5 subtasks | episodic accumulated return | Helped |
Limitations
- Cannot handle scenarios where confounders and selections co-occur
- Confounders tied to distribution shifts such as dynamics or embodiment are not addressed
- Seq-NMF is not designed for high-dimensional data; experiments use low-dimensional or compressed representations
Experimental setup
Datasets: Driving (self-collected) Synthetic Color-3 / Color-10 D4RL Kitchen demonstrations
Benchmarks: D4RL Kitchen (4-subtask and 5-subtask new tasks) Color-3 (Simple/Conditional) boundary detection Driving CI tests
Metrics: episodic accumulated return precision recall F1 CI test p-values
Baselines: H-AIRL Directed-info GAIL Option-GAIL VTA LOVE
Builds on: Option framework Generative Adversarial Imitation Learning Non-negative matrix factorization Directed-info GAIL Option-GAIL
Closest papers by meaning
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals (ICLR 2025 · similarity 0.86)
- Reward-Respecting Subtasks for Model-Based Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.86)
- BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning (ICLR 2026 · similarity 0.86)
- Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration (ICML 2025 · similarity 0.86)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.