Research map / Vision-language and multimodal / Cross-modal learning
Binding Touch to Everything: Learning Unified Multimodal Tactile Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Aligns a vision-based tactile encoder to ImageBind's frozen image embedding with learnable sensor-specific tokens and sensor-balanced batch sampling, enabling zero-shot touch understanding, cross-modal retrieval, touch-driven image synthesis, and Touch-LLM.
Problem
Cross-modal association between touch and other modalities is hard because tactile data is expensive to collect and vision-based touch sensors produce heterogeneous outputs, so existing tactile representations are typically limited to a single sensor.
Approach
Bind touch to the image embedding space of a pretrained multimodal model (ImageBind) via contrastive visuo-tactile alignment, so touch inherits links to text and audio without paired data; add learnable per-sensor prefix tokens and a sensor-aware in-batch sampling strategy to train on multiple sensors at once.
Key results
- Zero-shot material classification reaches 52.7% on Touch and Go, close to supervised baselines, and linear probing reaches 61.3% vs 49.2% for VT CMC (All).
- Cross-modal retrieval on ObjectFolder 2.0 achieves 41.9 mAP touch-to-vision, outperforming the fully supervised DAR (32.3).
- Touch-LLM gets a GPT-4 caption rating of 3.54 vs 2.33 for LLaVA-1.5 on 400 Touch and Go images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on multiple datasets with sensor-specific tokens and batch sampling rather than the baseline setup (ablation) | UniTouch · Touch and Go | accuracy (%): 21.4 -> 52.7 | Helped |
| Add sensor-specific tokens to multi-dataset training (ablation) | UniTouch · Touch and Go | accuracy (%): 21.4 -> 38.1 | Helped |
| Add the batch sampling strategy to multi-dataset training (ablation) | UniTouch · Touch and Go | accuracy (%): 21.4 -> 40.3 | Helped |
| Use the prompt “This feels like [CLS]” instead of “This is an image of [CLS]” (ablation) | UniTouch zero-shot material classification · Touch and Go | accuracy (%): 40.7 -> 52.7 | Helped |
| Use the prompt “This feels like [CLS]” instead of “This is an image of [CLS]” (ablation) | UniTouch zero-shot material classification · ObjectFolder 2.0 | accuracy (%): 34.3 -> 43.5 | Helped |
| Align touch embeddings to the pretrained multimodal image embedding | UniTouch · ObjectFolder 2.0 cross-sensory retrieval (touch → vision) | mAP: 32.3 -> 41.9 | Helped |
| Align touch embeddings to the pretrained multimodal image embedding | UniTouch · ObjectFolder 2.0 cross-sensory retrieval (touch → audio) | mAP: 27.8 -> 37.9 | Helped |
| Align touch embeddings to the pretrained multimodal image embedding | UniTouch · ObjectFolder 2.0 cross-sensory retrieval (touch → text) | mAP: 31.9 -> 38.0 | Helped |
| Use UniTouch instead of LLaVA-1.5 for touch image captioning | Touch-LLM · Touch and Go | GPT-4 Rating: 2.33 -> 3.54 | Helped |
Surprising findings
- Prompting with touch-specific wording such as 'This feels like' substantially improves zero-shot material classification over generic image prompts (52.7 vs 40.3 on Touch and Go).
- Naive multi-sensor training of the baseline drops performance from 43.1% to 21.4%, showing the severity of inter-sensor domain gaps.
Limitations
- Restricted to vision-based tactile sensors; other output formats (barometric, force) not covered.
- Learned representation is a black box with limited interpretability.
- Zero-shot audio and text retrieval are not trained with paired data, and the touch-to-text annotations are self-made.
Experimental setup
Datasets: Touch and Go Feeling of Success YCB-Slide ObjectFolder 2.0 ObjectFolder Real ObjectFolder 1.0 SSVTP
Benchmarks: Touch and Go ObjectFolder 2.0 YCB-Slide ObjectFolder 1.0 ObjectFolder Real SSVTP Feeling of Success
Metrics: accuracy mAP FID CVTP material classification consistency GPT-4 rating
Baselines: VT CMC SSVTP ImageNet supervised features CCA PLSCA DSCMR DAR Pix2Pix VisGel Vision-from-touch BLIP-2 InstructBLIP
Builds on: ImageBind CLIP VT CMC SSVTP
Closest papers by meaning
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.90)
- Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.88)
- A Touch, Vision, and Language Dataset for Multimodal Alignment (ICML 2024 · similarity 0.88)
- Universal Visuo-Tactile Video Understanding for Embodied Interaction (NeurIPS 2025 · similarity 0.87)
- Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper (NeurIPS 2025 · similarity 0.87)
- Sensor-Invariant Tactile Representation (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.