Research map / Vision-language and multimodal / Robot action and navigation
RDD: Retrieval-Based Demonstration Decomposer for Planner Alignment in Long-Horizon Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RDD, a training-free method that decomposes demonstrations into sub-tasks by retrieving visually similar intervals from the low-level visuomotor policy's training set, solved with dynamic programming, to finetune a VLM planner.
Problem
Hierarchical VLA planners are finetuned on demonstrations segmented by human annotation or heuristic rules, and these sub-tasks can deviate from the visuomotor policy's training distribution, degrading long-horizon task performance.
Approach
Formulate demonstration decomposition as an optimal partitioning problem whose interval score measures how close each proposed interval is (visually via embedding endpoints and temporally via duration) to its approximate nearest neighbor in the visuomotor policy's training set; solve exactly with a dynamic programming algorithm.
Key results
- On 13 RLBench tasks, RDD reaches 74.9% average success vs 71.4% for UVD and 75.1% for the expert heuristic decomposer.
- With LIV encoder over 3 seeds, RDD achieves 81.1% average success on the 13-task setting, and it reaches 0.706 IoU on AgiBotWorld real-world data vs 0.506 for UVD.
- Under OOD settings on RoboCerebra, RDD obtains IoU 0.630 (beta=0.10) vs 0.598 for UVD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| align planner demonstrations with the visuomotor policy training set using RDD instead of UVD | RACER · RLBench (13 tasks) | Avg. Succ. (%): 71.4 ± 5.1 -> 74.9 ± 6.9 | Helped |
| replace LIV with R3M as the visual encoder (ablation) | RDD · RLBench (13 tasks) | Avg. Succ. (%): 81.1 ± 0.9 -> 80.0 ± 3.5 | Hurt |
| replace LIV with VIP as the visual encoder (ablation) | RDD · RLBench (13 tasks) | Avg. Succ. (%): 81.1 ± 0.9 -> 75.3 ± 3.4 | Hurt |
| replace LIV with VC-1 as the visual encoder (ablation) | RDD · RLBench (13 tasks) | Avg. Succ. (%): 81.1 ± 0.9 -> 75.5 ± 3.1 | Hurt |
| replace LIV with CLIP as the visual encoder (ablation) | RDD · RLBench (13 tasks) | Avg. Succ. (%): 81.1 ± 0.9 -> 78.2 ± 2.1 | Hurt |
| replace LIV with DINOv2 as the visual encoder (ablation) | RDD · RLBench (13 tasks) | Avg. Succ. (%): 81.1 ± 0.9 -> 78.4 ± 2.4 | Hurt |
| replace LIV with ResNet as the visual encoder (ablation) | RDD · RLBench (13 tasks) | Avg. Succ. (%): 81.1 ± 0.9 -> 81.1 ± 2.5 | No clear effect |
| remove temporal alignment by setting α = 0 (ablation) | RDD · RLBench (13 tasks) | Avg. Succ.: 75.0 ± 2.5 -> 81.1 ± 0.9 | Hurt |
| set temporal weighting α = 0.5 instead of α = 1 (ablation) | RDD · RLBench (13 tasks) | Avg. Succ.: 75.7 ± 2.4 -> 81.1 ± 0.9 | Hurt |
| increase temporal weighting from α = 1 to α = 2 (ablation) | RDD · RLBench (13 tasks) | Avg. Succ.: 81.1 ± 0.9 -> 76.2 ± 3.0 | Hurt |
| use 1 demonstration per task instead of 3 (ablation) | RDD · RLBench (13 tasks) | Avg. Succ. (↑): 81.1 ± 0.9 -> 77.9 ± 4.5 | Hurt |
| replace RDD with UVD for 3 demonstrations per task (ablation) | RACER · RLBench (13 tasks) | Avg. Succ. (↑): 75.6 ± 1.8 -> 81.1 ± 0.9 | Hurt |
Surprising findings
- UVD performs similarly to naive uniform decomposition on RLBench, suggesting learned-feature change points are not aligned with the policy's training sub-tasks.
- Gemini-2.5-pro-based keyframe decomposition underperforms RDD (72.6% vs 74.9% success).
Limitations
- Retrieval operates on single-frame visual features and may not suit tasks requiring historical landmark images.
- Quality of decomposition depends on the quality of the visuomotor policy's training data; noisy skills would be matched too.
- OOD setting requires a hand-tuned trade-off hyperparameter beta and a general-subtask heuristic (UVD).
Experimental setup
Datasets: RLBench AgiBotWorld-Alpha RoboCerebra LIBERO
Benchmarks: RLBench AgiBotWorld-Alpha RoboCerebra LIBERO
Metrics: average success rate average rank mIoU IoU
Baselines: Expert heuristic decomposer UVD Uniform decomposition w/o Finetune Gemini-2.5-pro
Builds on: RACER UVD RVT LIV Annoy
Closest papers by meaning
- VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models (NeurIPS 2025 · similarity 0.88)
- ECHO: Continuous Hierarchical Memory for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.87)
- VICtoR: Learning Hierarchical Vision-Instruction Correlation Rewards for Long-horizon Manipulation (ICLR 2025 · similarity 0.87)
- Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization (NeurIPS 2025 · similarity 0.87)
- VLA-Pro: Cross-Task Procedural Memory Transfer for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.87)
- VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.