Research map / Vision-language and multimodal / Robot action and navigation
KISA: A Unified Keyframe Identifier and Skill Annotator for Long-Horizon Robotics Demonstrations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KISA, a temporal-enhanced pretrained vision-language representation trained with history-aware contrastive learning and monotonic distance loss to identify keyframes and annotate skills in long-horizon robot demonstrations.
Problem
Identifying keyframes and annotating semantic skills in unlabeled long-horizon robot manipulation demonstrations is inaccurate with frame-level visual distances or heuristics, which lack language grounding.
Approach
A lightweight Transformer temporal module over frozen-then-tuned pretrained robot representations (R3M/VIP/LIV) aggregates history frames, aligned to skill language embeddings via contrastive learning with hard negatives (incorrect skills, mismatched history, reversed video) plus a monotonic score-distance loss within sub-tasks.
Key results
- On CALVIN keyframe identification, KISA reaches 85.2% F1 versus 36.9% for UVD (LIV backbone) and 25.9% for LIV alone.
- Zero-shot on ManiSkill2 object generalization, KISA reaches 80.7% F1 vs 39.2% for UVD.
- On CALVIN policy learning with 1 instruction, LISA+KISA achieves 82.5% success vs 55.4% for LISA and 34.2% for LCBC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KISA instead of UVD for keyframe identification | LIV-based keyframe identification · Maniskill | F1 Score: 40.2 ± 1.1% -> 99.7 ± 0.2% | Helped |
| Use KISA instead of UVD for keyframe identification | LIV-based keyframe identification · CALVIN | F1 Score: 36.9 ± 0.5% -> 85.2 ± 0.9% | Helped |
| Use KISA instead of UVD for keyframe identification | LIV-based keyframe identification · FrankaKitchen | F1 Score: 64.8 ± 2.4% -> 98.7 ± 0.6% | Helped |
| Annotate demonstrations with KISA instead of using LISA's discovered skills | LISA policy learning · CALVIN (1 instruction) | Success rate: 55.4% ± 2.7 -> 82.5% ± 1.6 | Helped |
| Annotate demonstrations with KISA instead of using LISA's discovered skills | LISA policy learning · CALVIN (5 instructions) | Success rate: 9.3% ± 0.8 -> 17.6% ± 1.0 | Helped |
| Add monotonic alignment (ablation) | KISA-LIV · Maniskill2 | Top-1 Accuracy: 90.2 ± 0.3 % -> 99.2 ± 0.1% | Helped |
| Add monotonic alignment (ablation) | KISA-LIV · CALVIN | Top-1 Accuracy: 82.1 ± 0.4 % -> 94.7 ± 1.1% | Helped |
| Add monotonic alignment (ablation) | KISA-LIV · FrankaKitchen | Top-1 Accuracy: 89.1 ± 0.7% -> 96.1 ± 0.2% | Helped |
| Use history-aware contrastive learning instead of vanilla contrastive learning (ablation) | KISA-LIV · Maniskill2 | Top-1 Accuracy: 59.1 ± 1.9 % -> 99.2 ± 0.1% | Helped |
| Use history-aware contrastive learning instead of vanilla contrastive learning (ablation) | KISA-LIV · CALVIN | Top-1 Accuracy: 58.1 ± 1.3 % -> 94.7 ± 1.1% | Helped |
| Use history-aware contrastive learning instead of vanilla contrastive learning (ablation) | KISA-LIV · FrankaKitchen | Top-1 Accuracy: 73.7 ± 0.4% -> 96.1 ± 0.2% | Helped |
| Add temporal enhancement (ablation) | KISA-LIV · Maniskill2 | Top-1 Accuracy: 22.0 ± 1.6 % -> 99.2 ± 0.1% | Helped |
Surprising findings
- Annotations from the raw LIV representation degrade policy learning below plain LISA (LISA+LIV 51.2% vs LISA 55.4% on single-instruction tasks), showing low-quality annotations can hurt.
Limitations
- Skill annotation retrieves from a fixed skill library and cannot generate open-vocabulary skill descriptions.
- Lower accuracy on the real-kitchen embodiment transfer setting (F1 40.7%)
- Joint training across domains slightly degrades performance relative to per-domain models.
Experimental setup
Datasets: ManiSkill2 CALVIN Franka Kitchen Ego-style long-horizon kitchen demonstrations (Xu et al., 2023)
Benchmarks: ManiSkill2 CALVIN Franka Kitchen real kitchen demonstrations
Metrics: keyframe number error MAE F1 score Top-1 accuracy
Baselines: KTS VideoRLCS UVD R3M VIP LIV LCBC LISA
Builds on: LIV CLIP UVD LISA
Closest papers by meaning
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.88)
- VICtoR: Learning Hierarchical Vision-Instruction Correlation Rewards for Long-horizon Manipulation (ICLR 2025 · similarity 0.87)
- DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation (NeurIPS 2026 · similarity 0.87)
- Robotic Visual Instruction (CVPR 2025 · similarity 0.86)
- Beyond World-Frame Action Heads: Motion-Centric Action Frames for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.86)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.