Research map / Vision-language and multimodal / Human and embodied modeling
Interactive Visual Task Learning for Robots
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hi-Viscont, a hierarchical visual concept learner that propagates novel-concept information to parent concepts and uses scene graphs with language annotations so a robot can learn visual tasks one-shot and generalize zero-shot, evaluated on VQA and a 18-participant user study against FALCON.
Problem
Robots need to learn novel visual concepts and visual tasks one-shot from in-situ linguistic interaction, and existing concept learners do not update parent concepts when new leaf concepts are added, which hurts continual learning.
Approach
Augments a novel concept's information to its ancestor nodes via an Ancestor Relational GNN (ARGNN) so the whole concept hierarchy updates continually, and represents demonstrated tasks as language-annotated scene graphs that allow zero-shot generation of task variants.
Key results
- On non-leaf concepts Hi-Viscont significantly outperforms FALCON in the Zoo domain (85.84 vs 66.15 F1) and on CUB genera/family/order, while leaf-level performance is comparable.
- In an 18-participant human study, Hi-Viscont improves success rate by 33.33% over FALCON (Wilcoxon p=0.014) and node-level accuracy by 19.44% (p=0.005).
- Users significantly preferred Hi-Viscont in the comparative survey (Z=2.0, p<0.001) and rated it higher on trust, SUS, perceived intelligence, and anthropomorphism.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Update related ancestor concept representations when adding a novel concept with ARGNN | Hi-Viscont · CUB-200-2011 Genera, Family, and Order concepts | F1: Genera: 88.2±1.0 -> 90.4±0.6; Family: 84.3±1.4 -> 90.7±1.7; Order: 84.3±3.2 -> 92.0±0.8 | Helped |
| Update related ancestor concept representations when adding a novel concept with ARGNN | Hi-Viscont · CUB-200-2011 Species concepts | F1: 86.5±1.4 -> 87.1±2.0 | No clear effect |
| Update related ancestor concept representations when adding a novel concept with ARGNN | Hi-Viscont · House Construction Color concepts | F1: 87.27±5.83 -> 99.24±0.70 | Helped |
| Update related ancestor concept representations when adding a novel concept with ARGNN | Hi-Viscont · House Construction Affordance concepts | F1: 57.35±9.23 -> 89.86±9.12 | Helped |
| Update related ancestor concept representations when adding a novel concept with ARGNN | Hi-Viscont · House Construction Object concepts | F1: 89.28±0.93 -> 88.46±1.58 | No clear effect |
| Update related ancestor concept representations when adding a novel concept with ARGNN | Hi-Viscont · Zoo Non-leaf concepts | F1: 66.15±5.34 -> 85.84±5.79 | Helped |
| Update related ancestor concept representations when adding a novel concept with ARGNN | Hi-Viscont · Zoo Leaf concepts | F1: 88.99±3.75 -> 87.93±3.40 | No clear effect |
| Use Hi-Viscont instead of FALCON for in-situ task learning | Hi-Viscont · House Construction human-subjects task | Success Rate (%): 16.67±38.25 -> 50.00±51.45; 33.33% improvement | Helped |
| Use Hi-Viscont instead of FALCON for in-situ task learning | Hi-Viscont · House Construction human-subjects task | Node Accuracy (%): 61.81±23.67 -> 81.25±21.11; 19.44% improvements | Helped |
Surprising findings
- On the House domain FALCON slightly exceeds Hi-Viscont on object (leaf) F1 (89.28 vs 88.46), and on the CUB Species level both models are essentially tied.
Limitations
- Robot study limited to the House domain with a small number of objects.
- Method does not transfer automatically to new domains and requires training the concept net from scratch.
- User interaction follows controlled turn-taking rules and the sample consists of college-age participants.
Experimental setup
Datasets: CUB-200-2011 House Construction (custom) Zoo (custom)
Benchmarks: CUB-200-2011 VQA House Construction VQA Zoo VQA human-subjects house construction study
Metrics: F1 precision recall success rate node-level accuracy Godspeed perceived intelligence/anthropomorphism trust SUS comparative preference
Baselines: FALCON
Builds on: FALCON box embedding Segment Anything Model BERT
Closest papers by meaning
- PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs (ICML 2024 · similarity 0.88)
- ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Tasks (NeurIPS 2025 · similarity 0.87)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.87)
- JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics (AAAI 2026 · similarity 0.87)
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.87)
- ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.