Research map / Continual learning / Domain and prompt-based
Think Small, Act Big: Primitive Prompt Learning for Lifelong Robot Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Primitive Prompt Learning (PPL), which learns motion-aware (optical flow + CLIP text) primitive prompts during multi-skill pretraining of a diffusion transformer policy and then concatenates frozen-plus-new lifelong prompts for continual robot skill acquisition.
Problem
Lifelong robot manipulation needs to transfer knowledge across skills and avoid catastrophic forgetting, but experience replay and parameter-efficient methods fail to exploit primitives shared between semantically different skills.
Approach
Prefix prompts for a diffusion transformer policy are queried by a text-plus-optical-flow (motion-aware) key, so semantically distinct skills that share motion primitives activate common prompts; in the lifelong stage, new prompt components are appended while pretrained ones stay frozen.
Key results
- On MimicGen lifelong learning, PPL reaches average FWT 0.78 ± 0.09 versus 0.61 for LoRA and 0.65 for sequential fine-tuning, and BWT-style retention is reported for baselines.
- In real-world pretraining, PPL averages 0.84 success across two pretrain tasks versus 0.73 for MoE and 0.42 for Diffusion Transformer.
- On LIBERO-GOAL pretraining, PPL averages 0.84 versus 0.82 for MoE and 0.77 for Diff-T, and the text-flow query yields lower performance than text-only under extreme lighting (0.61 vs 0.80 at L3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PPL instead of LoRA | Lifelong learning on MimicGen · MimicGen, Average | FWT: 0.78 ± 0.09 -> 0.83 ± 0.03 | Helped |
| use PPL instead of Sequential | Lifelong learning on real-world robot tasks · Real-world, Average | success rate: 0.55 ± 0.09 -> 0.68 ± 0.05 | Helped |
| use PPL instead of MOE | Multi-skill pre-training on real-world robot tasks · Real-world, Average | success rate: 0.73 ± 0.08 -> 0.84 ± 0.05 | Helped |
| use text-only rather than flow+text prompt query (ablation) | PPL under changing lighting · Lighting L3 (warm→cold→dark) | success rate: 0.61 -> 0.80 | Helped |
| use text-only rather than flow+text prompt query (ablation) | PPL under lighting conditions · Static dark light | success rate: 0.48 -> 0.61 | Helped |
| use text-only rather than flow+text prompt query (ablation) | PPL under lighting conditions · Lighting L1 (warm) | success rate: 0.83 -> 0.81 | Hurt |
| use text-only rather than flow+text prompt query (ablation) | PPL under lighting conditions · Static cold light | success rate: 0.87 -> 0.86 | Hurt |
| remove primitive prompt learning during pre-training (ablation) | PPL lifelong skill acquisition · New skills | success rate | Hurt |
| remove pretrained prompts during new-skill acquisition (ablation) | PPL lifelong skill acquisition · New skills | success rate | Hurt |
| increase the number of primitive prompts (ablation) | PPL multi-skill learning · Multi-skill pre-training | success rate | Mixed |
| use text-flow query instead of text-only query (ablation) | PPL prompt weighting · Semantically distinct skills | prompt-weight similarity | Helped |
| use optical-flow MAP (ablation) | PPL under lighting variation · Lighting L3 (warm→cold→dark) | success rate: 0.75 -> 0.61 | Hurt |
Surprising findings
- Under extreme dynamic lighting (warm to cold to dark), the text-plus-flow query performs worse than a text-only query (0.61 vs 0.80), contrary to the motivation for flow-based prompting.
- The authors report that MoE achieves higher average success than PPL in some settings, while PPL is faster to train.
Limitations
- Performance degrades under strong lighting changes; flow-based query underperforms text-only in that setting.
- Performance is sensitive to the number of primitive prompts; more prompts do not necessarily help.
- Evaluation uses a self-constructed skill dataset and small numbers of tasks (4 real-world lifelong tasks, 7 simulated MimicGen tasks).
Experimental setup
Datasets: MimicGen LIBERO
Benchmarks: MimicGen LIBERO-GOAL Real-world Franka tasks
Metrics: success rate FWT BWT
Baselines: Sequential fine-tuning Experience replay (ER) LoRA TAIL MoE policy Diffusion Transformer policy
Builds on: Diffusion Policy RAFT CLIP LoRA prefix-tuning DualPrompt
Closest papers by meaning
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning (NeurIPS 2023 · similarity 0.91)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.87)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.87)
- QueST: Self-Supervised Skill Abstractions for Learning Continuous Control (NeurIPS 2024 · similarity 0.87)
- RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation (ICML 2024 · similarity 0.87)
- Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.