Research map / Vision-language and multimodal / Human and embodied modeling
Tools Identification By On-Board Adaptation of Vision-and-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents an on-board online domain adaptation system that adapts CLIP with hyperdimensional computing so a small edge device can identify workshop tools from explicit and purpose-driven audio queries.
Problem
Off-the-shelf vision-and-language models trained on internet images fail to identify small, obscure tools in cluttered scenes under domain shift, and cannot match tools to their purpose or affordances, for a robotic workshop assistant.
Approach
Two-stage pipeline (region proposals then a classifier) where CLIP embeddings of ROI crops and of tool-name or purpose captions are randomly projected to binary hyperdimensional vectors, bound with XOR and bundled into class exemplars, enabling fast online adaptation on an edge processor.
Key results
- Zero-shot mAP transferring from SKIML to ALET rises from 9.5 with vision only to 20.5 with vision+language; ALET to SKIML rises from 5 to 20.
- Zero-shot transfer from 12 SKIML tools to 50 ALET tools reaches about 20 mAP versus about 40 mAP when finetuned on the target domain.
- Purpose-based exemplars raise mAP@0.5 from 27.61 (name exemplars) to 43.57 on purpose queries.
- On NVIDIA Orin, total per-image time is about 27 s (18.08 s region proposals, 6.85 s CLIP, 1.95 s HD), versus about 7 s on an A5000 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add language to vision | CLIP zero-shot detection · ALET to SKIML | mAP: 9.5 -> 20.5 | Helped |
| Add language to vision | CLIP zero-shot detection · SKIML to ALET | mAP: 5 -> 20 | Helped |
| Use purpose annotations for exemplars instead of tool names (ablation) | CLIP tool identification · Purpose queries | mAP@0.5: 27.61 -> 43.57 | Helped |
| Use purpose annotations for exemplars instead of tool names (ablation) | CLIP tool identification · Purpose queries | mAP@0.5:0.95: 18.43 -> 28.83 | Helped |
Surprising findings
- Adding purpose-driven text to the classifier roughly doubles zero-shot mAP in both transfer directions.
Limitations
- Region proposal generation dominates runtime on the edge device.
- SKIML dataset is proprietary and not released.
- Zero-shot performance remains about half of finetuned mAP on the target domain.
Experimental setup
Datasets: ALET SKIML in-house English speech data (2250 hours)
Benchmarks: ALET SKIML
Metrics: mAP mAP@0.5 mAP@0.5:0.95 runtime per image
Baselines: vision-only zero-shot detection finetuned detector on target domain
Builds on: CLIP Hyperdimensional Computing Faster R-CNN region proposal network
Closest papers by meaning
- Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.85)
- Thinking with Programming Vision: Towards a Unified View for Thinking with Images (CVPR 2026 · similarity 0.85)
- Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.85)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.83)
- Task Me Anything (NeurIPS 2024 · similarity 0.83)
- AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoning (ICLR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.