Research map / Vision-language and multimodal / Human and embodied modeling
A Touch, Vision, and Language Dataset for Multimodal Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the TVL dataset of ~44K in-the-wild touch-vision pairs with human (10%) and GPT-4V pseudo-labels, trains a vision-language-aligned tactile encoder and a TVL-LLaMA model for tactile description generation.
Problem
Touch is missing from multimodal language models, largely because tactile data lacks natural language labels and tactile readings are hard to align jointly with vision and language.
Approach
Pairwise contrastive alignment across all three modalities (tactile-vision, tactile-language, vision-language) instead of ImageBind-style binding to vision, with GPT-4V pseudo-labels on in-the-wild handheld-collected data, and a LLaMA-2 model finetuned on top of the tactile encoder via the ImageBind-LLM recipe.
Key results
- TVL tactile encoder (ViT-Tiny) reaches 36.7% top-1 tactile-text and 79.5% top-1 tactile-vision accuracy on the 402-way test, versus 28.4% vision-text for OpenCLIP and 0.2% tactile-vision for SSVTP.
- TVL-LLaMA (ViT-Base) scores 5.03 on the TVL Benchmark overall versus 4.49 for GPT-4V and 1.24-3.80 for other open VLMs, with GPT-4 scoring 1-10.
- Abstract reports +29% classification accuracy improvement and +12% over GPT-4V on the TVL benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train TVL-LLaMA with the TVL tactile encoder and human-labeled examples | TVL-LLaMA · TVL Benchmark | score: 4.49 -> 5.03 | Helped |
| add direct tactile-text contrastive loss (ablation) | TVL encoder · TVL tactile-text classification | top-1 accuracy: 20.3 -> 36.3 | Helped |
| add direct tactile-text contrastive loss (ablation) | TVL encoder · TVL tactile-vision classification | top-1 accuracy: 81.6 -> 78.0 | Hurt |
| include vision in contrastive training (ablation) | TVL encoder · TVL tactile-vision classification | top-1 accuracy: 1.0 -> 78.0 | Helped |
| include vision in contrastive training (ablation) | TVL encoder · TVL tactile-text classification | top-1 accuracy: 29.9 -> 36.3 | Helped |
| include text in contrastive training (ablation) | TVL encoder · TVL tactile-text classification | top-1 accuracy: 21.5 -> 36.3 | Helped |
| include text in contrastive training (ablation) | TVL encoder · TVL tactile-vision classification | top-1 accuracy: 85.8 -> 78.0 | Hurt |
| add no-contact frames to training (ablation) | TVL encoder · TVL tactile-text classification | top-1 accuracy: 36.2 -> 36.3 | No clear effect |
| add no-contact frames to training (ablation) | TVL encoder · TVL tactile-vision classification | top-1 accuracy: 80.1 -> 78.0 | Hurt |
| add CLIP-style prompting (ablation) | TVL encoder · TVL tactile-text classification | top-1 accuracy: 36.3 -> 37.7 | Helped |
| add CLIP-style prompting (ablation) | TVL encoder · TVL tactile-vision classification | top-1 accuracy: 78.0 -> 78.7 | Helped |
| train on HCT rather than SSVTP (ablation) | TVL encoder · TVL tactile-vision classification | top-1 accuracy: 8.0 -> 74.4 | Helped |
Surprising findings
- Disabling the tactile-language loss drops tactile-text top-1 accuracy from 36.3% to 20.3%, while tactile-vision accuracy slightly rises.
- Models trained on the GPT-4V pseudo-labeled HCT subset alone (38.4% tactile-text) match or exceed the full TVL mix, suggesting pseudo-labels are sufficient for the encoder.
Limitations
- GPT-4V pseudo-labels are limited by vision-only observation; occlusion of the contact patch makes labels inaccurate.
- Overfitting to noisy GPT-4V pseudo-labels on contact-only data required adding non-contact frames, and a distribution shift exists between pseudo-labeled training and human-labeled test sets.
- Only 10% of the data is human-labeled, and the benchmark relies on GPT-4 judging against human labels.
Experimental setup
Datasets: TVL (SSVTP + HCT) SSVTP LLaVA Visual Instruct CC3M 595K Alpaca LLaVA Visual Instruct 150K
Benchmarks: TVL Benchmark TVL open-vocabulary tactile classification (402-way)
Metrics: top-1 accuracy top-5 accuracy GPT-4 similarity score (1-10) p-value (paired t-test)
Baselines: OpenCLIP SSVTP GPT-4V LLaVA-1.5 ViP-LLaVA LLaMA-Adapter BLIP-2 InstructBLIP SSVTP-LLaMA
Builds on: ImageBind ImageBind-LLM OpenCLIP LLaMA-Adapter SSVTP
Closest papers by meaning
- TextToucher: Fine-Grained Text-to-Touch Generation (AAAI 2025 · similarity 0.89)
- Universal Visuo-Tactile Video Understanding for Embodied Interaction (NeurIPS 2025 · similarity 0.89)
- Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.88)
- Binding Touch to Everything: Learning Unified Multimodal Tactile Representations (CVPR 2024 · similarity 0.88)
- STOLA: Self-Adaptive Touch-Language Framework for Tactile Commonsense Reasoning in Open-Ended Scenarios (AAAI 2026 · similarity 0.87)
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.