Research map / Multi-agent systems and games / Human-AI and goal-conditioned
NBDI: A Simple and Effective Termination Condition for Skill Extraction from Task-Agnostic Demonstrations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NBDI, which terminates variable-length skills at state-action novelty points (ICM prediction error) learned from task-agnostic demonstrations, improving downstream long-horizon RL over fixed-length SPiRL.
Problem
Fixed-length skills in skill-based RL skip critical decision points, limiting exploration and policy learning in long-horizon sparse-reward tasks; existing variable-length skill methods scale poorly or fail to transfer to changed environments.
Approach
Identify skill termination points as states with high joint state-action novelty (decomposed into state novelty times conditional action novelty), estimated with a pre-trained ICM on task-agnostic offline data, and use these learned termination conditions in a SMDP-based skill-prior RL framework.
Key results
- NBDI achieves 0.25 success on Maze 40x40 vs 0.09 for SPiRL (177.78% improvement) and 0.24 vs 0.13 on Maze 30x30.
- On Kitchen NBDI completes 3.67 subtasks vs 3.0 for SPiRL (22.33% improvement); SAC and BC+SAC fail to make progress.
- On sparse block stacking NBDI stacks 1.12 blocks vs 0.67 for SPiRL; in SiMPL meta-RL, SiMPL+NBDI reaches 0.560 vs 0.143 success at ep20.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use state-action novelty-based variable-length skills instead of fixed-length skills | SPiRL-style skill learning · Maze 30x30 | success rate: 0.13$\pm$0.03 -> \textbf{0.24}$\pm$0.01 | Helped |
| Use state-action novelty-based variable-length skills instead of fixed-length skills | SPiRL-style skill learning · Maze 40x40 | success rate: 0.09$\pm$0.02 -> \textbf{0.25}$\pm$0.02 | Helped |
| Use state-action novelty-based variable-length skills instead of fixed-length skills | SPiRL-style skill learning · Sparse Block Stacking | stacked blocks: 0.67$\pm$0.29 -> \textbf{1.12}$\pm$0.16 | Helped |
| Use state-action novelty-based variable-length skills instead of fixed-length skills | SPiRL-style skill learning · Kitchen | completed subtasks: 3.0$\pm$0.0 -> \textbf{3.67}$\pm$0.43 | Helped |
| Use NBDI termination with SiMPL | SiMPL · Maze meta-RL target task, ep20 | success rate: 0.143$\pm$0.063 -> \textbf{0.560}$\pm$0.121 | Helped |
| Use NBDI termination with SiMPL | SiMPL · Maze meta-RL target task, ep100 | success rate: 0.593$\pm$0.191 -> \textbf{0.960}$\pm$0.021 | Helped |
| Use NBDI termination with SiMPL | SiMPL · Maze meta-RL target task, ep300 | success rate: 0.667$\pm$0.193 -> \textbf{0.980}$\pm$0.011 | Helped |
| Use NBDI termination with SiMPL | SiMPL · Maze meta-RL target task, ep500 | success rate: 0.990$\pm$0.006 -> \textbf{0.993}$\pm$0.003 | Helped |
| Add conditional action novelty to state novelty for decision-point identification (ablation) | NBDI · Downstream tasks | policy learning performance | Helped |
| Set an appropriate novelty threshold rather than using an unsuitable threshold (ablation) | NBDI · Kitchen | policy performance | Helped |
| Learn a termination distribution during offline skill extraction (ablation) | NBDI-NoTermDistr · Downstream tasks | policy performance | Helped |
| Use thresholded state-action novelty rather than cumulative-sum novelty to determine terminations (ablation) | NBDI-CumulativeSum · Downstream tasks | policy performance | Helped |
Surprising findings
- Terminating on state novelty alone does not improve performance; the benefit comes from combining it with conditional action novelty.
- Performance degrades when the offline dataset is generated by highly stochastic policies or random walks, so decision points become hard to detect.
Limitations
- Sensitive to dataset stochasticity; fails with random-walk data.
- Novelty threshold must be tuned per environment (0.3, 50, 40).
- Relies on ICM which may scale poorly to very large or continuous state spaces without cropping.
Experimental setup
Datasets: D4RL kitchen task-agnostic maze demonstrations task-agnostic sparse block stacking demonstrations MuJoCo D4RL halfcheetah and ant (visualization only)
Benchmarks: Maze 30x30 Maze 40x40 Sparse Block Stacking Kitchen
Metrics: success rate stacked blocks completed subtasks return
Baselines: SAC SAC+Novelty BC+SAC IQL+SAC SPiRL LOVE Relative Novelty SiMPL
Builds on: SPiRL option framework intrinsic curiosity module Soft Actor-Critic termination improvement theorem
Closest papers by meaning
- Efficient Skill Discovery via Regret-Aware Optimization (ICML 2025 · similarity 0.86)
- A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals (ICLR 2025 · similarity 0.86)
- Exploration by Learning Diverse Skills through Successor State Representations (NeurIPS 2024 · similarity 0.86)
- Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment (ICLR 2025 · similarity 0.86)
- Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
- Skill Disentanglement in Reproducing Kernel Hilbert Space (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.