Research map / Vision-language and multimodal / Multimodal LLMs
Universal Visuo-Tactile Video Understanding for Embodied Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VTV-LLM, a Qwen2.5-based multimodal LLM for visuo-tactile video understanding, trained in three stages on a new 150K-frame multi-sensor dataset (VTV150K) with tactile attribute annotations.
Problem
Multimodal LLMs cannot perceive tactile attributes (hardness, protrusion, elasticity, friction) from visuo-tactile sensor video, limiting embodied physical reasoning that vision alone cannot support.
Approach
A VTV encoder pretrained with optical-flow-guided masking and an attribute classifier, then aligned to a Qwen2.5 LLM through projectors and finetuned on templated visuo-tactile QA pairs.
Key results
- VTV-LLM-7B reaches 60.4% average accuracy versus 28.0% for GPT-4o and 29.5% for Gemini-2.5-Pro-Exp on the VTV150K test set.
- On tactile attribute assessment, VTV-LLM-7B scores 73.9% hardness and 35.6% combined, far above the best baseline.
- Removing stage 3 (text prompt finetuning) drops average accuracy from 59.0% to 43.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train VideoMAE with VTV150K rather than leave it untrained (ablation) | VTV-LLM VTV encoder · VTV150K SFD, SOI, OSC, TSA | Average: 22.9 -> 41.2 | Helped |
| Use optical flow-guided masking instead of a trained VideoMAE encoder (ablation) | VTV-LLM VTV encoder · VTV150K SFD, SOI, OSC, TSA | Average: 41.2 -> 51.5 | Helped |
| Add the attribute classifier and joint attribute-classification loss (ablation) | VTV-LLM VTV encoder · VTV150K SFD, SOI, OSC, TSA | Average: 51.5 -> 59.0 | Helped |
| Add stage 2 VTV-text alignment (ablation) | VTV-LLM three-stage training · VTV150K SFD, SOI, OSC, TSA | Average: 50.8 -> 59.0 | Helped |
| Add stage 3 text prompt finetuning (ablation) | VTV-LLM three-stage training · VTV150K SFD, SOI, OSC, TSA | Average: 43.5 -> 59.0 | Helped |
| Use independent datasets across training stages rather than the same dataset (ablation) | VTV-LLM three-stage training · VTV150K SFD, SOI, OSC, TSA | Average: 51.7 -> 59.0 | Helped |
| Increase Qwen 2.5 backbone size from 3B to 7B to 14B (ablation) | VTV-LLM · VTV150K, especially TSA | Performance | Helped |
Surprising findings
- Frontier proprietary models GPT-4o and Gemini-2.5-Pro score only about 28-30% average, barely above the open-source 7B VLMs.
Limitations
- Evaluation relies on a single self-constructed template-based benchmark with a small 600-question test set.
- Tactile attribute labels are coarse three-level categories and the dataset covers only 100 objects.
- Reported results of 59.0 vs 60.4 average are inconsistent between the ablation and main table.
Experimental setup
Datasets: VTV150K
Benchmarks: VTV150K test set (600 QA pairs)
Metrics: accuracy (%)
Baselines: GPT-4o Gemini-2.5-Pro-Exp LLaVA-OneVision-7B LLaVA-Video-Qwen2-7B InternVL2.5-VL-8B VideoLLaMA3-7B Qwen2.5-VL-7B VideoMAE
Builds on: VideoMAE RAFT Qwen2.5 VideoMAEv2
Closest papers by meaning
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.89)
- Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.89)
- A Touch, Vision, and Language Dataset for Multimodal Alignment (ICML 2024 · similarity 0.89)
- Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper (NeurIPS 2025 · similarity 0.88)
- Sensor-Invariant Tactile Representation (ICLR 2025 · similarity 0.88)
- Binding Touch to Everything: Learning Unified Multimodal Tactile Representations (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.