Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Few-Shot Task Learning through Inverse Generative Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FTL-IGM, which learns new task concepts from a few unlabeled demonstrations by inverting a frozen pretrained conditional diffusion model, and composes them with training concepts across five domains.
Problem
Few-shot learning of task concepts (goals or motion styles) from a handful of unlabeled agent demonstrations, with generalization to new initial states and composition with training concepts.
Approach
Freeze a conditional diffusion model pretrained on (behavior, concept) pairs and optimize new concept embeddings (and composition weights) to maximize the likelihood of new demonstrations, so the learned concepts can be reused for new initial states and compositions without finetuning weights or acting in the environment.
Key results
- Object rearrangement new-concept accuracy 0.82 ± 0.09 for FTL-IGM vs 0.08 for VAE and 0.67 for BC (new concepts/compositions from new initial states, per Table).
- Human MoCap study: FTL-IGM selected as most successful in 80% of training-concept scenarios and 73.3% of new-concept scenarios, versus 20%/26.7% for BC and 0%/0% for VAE.
- Table-top manipulation: learned new concept 'elevated pushing' reaches 55% accuracy vs 15% for using the training push representation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn 2 concept components and weights instead of 1 (ablation) | FTL-IGM · Object rearrangement training composition | accuracy: 0.28±0.11 -> 0.75±0.11 | Helped |
| learn 2 concept components and weights instead of 1 (ablation) | FTL-IGM · Object rearrangement new concept | accuracy: 0.49±0.11 -> 0.77±0.09 | Helped |
| learn 2 concept components and weights instead of 1 (ablation) | FTL-IGM · Object rearrangement new and training concept composition | accuracy: 0.38±0.04 -> 0.73±0.04 | Helped |
| learn 2 concept components and weights instead of 1 (ablation) | FTL-IGM · AGENT training composition | accuracy: 0.52±0.08 -> 0.68±0.13 | Helped |
| learn 2 concept components and weights instead of 1 (ablation) | FTL-IGM · AGENT new initial state | accuracy: 0.54±0.07 -> 0.67±0.05 | Helped |
| learn 2 concept components and their weights instead of 1 (ablation) | FTL-IGM · Driving new initial state | accuracy: 0.14 -> 0.24 | Helped |
| use classifier-free guidance weight 1.2 rather than 1.8 (ablation) | FTL-IGM · Object rearrangement new concepts on new initial states | accuracy: 0.61 ± 0.06 -> 0.82 ± 0.09 | Helped |
| use classifier-free guidance weight 1.6 rather than 1.8 (ablation) | FTL-IGM · AGENT new concepts on new initial states | accuracy: 0.63 ± 0.06 -> 0.73 ± 0.07 | Helped |
| learn classifier-free guidance weights instead of using fixed guidance weights (ablation) | FTL-IGM · Driving roundabout, closed loop | success rate: 18% -> 24%; 14% -> 24%; 4% -> 24%; 10% -> 24% | Helped |
| learn classifier-free guidance weights instead of using fixed guidance weights (ablation) | FTL-IGM · Driving roundabout, closed loop | crash rate: 38% -> 32%; 40% -> 32%; 48% -> 32%; 40% -> 32% | Mixed |
| learn the new task concept from demonstrations rather than reuse a training representation | FTL-IGM · Real-world table-top manipulation, pushing on book | success rate: 15% -> 55% | Helped |
Surprising findings
- Learning two concept components with weights outperforms learning a single concept in object rearrangement and AGENT (e.g., 0.28 to 0.75 for training compositions).
- Language conditioning with T5 embeddings of new concept descriptions largely fails on new concepts (e.g., 0.2 accuracy in object rearrangement), and for MoCap the agent transitions into walking.
Limitations
- Diffusion model incurs high inference time.
- Learned concepts are assumed to lie within the span of training concepts; no theoretical analysis of which new concepts are representable.
- Generation quality on MoCap is noisy and demonstration compatibility of composed concepts is imperfect.
Experimental setup
Datasets: CMU Graphics Lab Motion Capture Database AGENT HighwayEnv Franka teleop table-top demonstrations (self-collected) Synthetic object rearrangement data (self-generated)
Benchmarks: Object rearrangement (synthetic 2D) AGENT goal-oriented navigation CMU MoCap human action generation HighwayEnv autonomous driving Franka table-top manipulation
Metrics: accuracy crash rate task completion (success) rate human preference selection rate
Baselines: goal-conditioned behavior cloning VAE In-Context learning (Xu et al.) Language conditioning
Builds on: classifier-free guidance Diffuser Composable diffusion Textual inversion AGENT
Closest papers by meaning
- Solving New Tasks by Adapting Internet Video Knowledge (ICLR 2025 · similarity 0.86)
- VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought (NeurIPS 2024 · similarity 0.86)
- OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation (NeurIPS 2025 · similarity 0.85)
- QueST: Self-Supervised Skill Abstractions for Learning Continuous Control (NeurIPS 2024 · similarity 0.85)
- RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation (ICML 2024 · similarity 0.85)
- InfoCon: Concept Discovery with Generative and Discriminative Informativeness (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.