Research map / Vision-language and multimodal / Cross-modal learning
AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Collects TacQuad, an aligned multi-modal multi-sensor tactile dataset from four visuo-tactile sensors, and trains AnyTouch, a static-dynamic unified tactile representation model using masked modeling, touch-vision-language alignment, cross-sensor matching, and universal sensor tokens.
Problem
Visuo-tactile sensors are poorly standardized, so data from different sensors differ, hindering a unified tactile representation and cross-sensor transfer to new sensors and tasks.
Approach
Learn a unified multi-sensor tactile representation from both static tactile images and dynamic tactile videos via a two-stage multi-level scheme (masked reconstruction, then touch-vision-language alignment plus cross-sensor matching), using aligned multi-sensor data and universal sensor tokens that substitute for sensor-specific tokens.
Key results
- AnyTouch achieves 82.74 material accuracy on Touch and Go vs 67.45 for OmniBind and 61.3 for UniTouch (trained on same data).
- On unseen sensor OF 1.0 material classification, AnyTouch with full data reaches 49.62 vs 41.0 for CLIP and 47.25 for UniTouch†.
- In the real-world pouring task, AnyTouch achieves the best mean error over T3 and a static-only AnyTouch baseline (exact values in table not extracted).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text modality for alignment (ablation) | AnyTouch · TAG material | accuracy: 75.91 -> 80.82 | Helped |
| add vision modality for alignment (ablation) | AnyTouch · TAG material | accuracy: 74.55 -> 80.82 | Helped |
| include text descriptions in TacQuad (ablation) | AnyTouch · TAG material | accuracy: 80.70 -> 80.82 | Helped |
| add Stage 1 masked modeling (ablation) | AnyTouch · TAG material | accuracy: 78.34 -> 80.82 | Helped |
| add Stage 2 semantic alignment and matching (ablation) | AnyTouch · TAG material | accuracy: 68.64 -> 80.82 | Helped |
| add cross-sensor matching (ablation) | AnyTouch · TAG material | accuracy: 80.54 -> 80.82 | Helped |
| add dynamic perception training (ablation) | AnyTouch · TAG material | accuracy: 77.93 -> 80.82 | Helped |
| add universal sensor tokens (ablation) | AnyTouch · TAG material | accuracy: 80.79 -> 80.82 | No clear effect |
| add data from more sensors to GelSight pretraining | AnyTouch · TAG | material accuracy: 83.55 -> 79.67 | Hurt |
| add cross-sensor matching (ablation) | AnyTouch · ObjectFolder 1.0 | material accuracy: 49.25 -> 49.62 | Helped |
| add universal sensor tokens (ablation) | AnyTouch · Feel | grasp accuracy: 79.03 -> 80.53 | Helped |
| add joint dynamic-perception training (ablation) | AnyTouch · ObjectFolder 2.0 | material accuracy: 75.70 -> 76.02 | Helped |
Surprising findings
- Adding more sensors' data slightly reduced accuracy on the seen TAG material task, attributed to reduced TAG share in pretraining data.
- Stage-1-only ablation of stage 2 removal drops OF 2.0 below CLIP, showing semantic-level learning is crucial for unseen sensors.
Limitations
- TacQuad is limited in scale and object variety compared to total training data.
- Only four visuo-tactile sensor types are covered, and intra-type sensor differences are not explored.
- Dynamic perception is validated on a single real-world task (pouring).
Experimental setup
Datasets: Touch and Go VisGel Cloth ObjectFolder Real TVL YCB-Slide SSVTP Octopi TacQuad Feel ObjectFolder 1.0 ObjectFolder 2.0
Benchmarks: Touch and Go Feel ObjectFolder 1.0 ObjectFolder 2.0 real-world fine-grained pouring
Metrics: classification accuracy mean error (pouring mass) mean squared error (cross-sensor generation)
Baselines: CLIP VIT-LENS-2 TLV-Link OmniBind UniTouch T3
Builds on: OpenCLIP masked autoencoder UniTouch TLV-Link LoRA
Closest papers by meaning
- Sensor-Invariant Tactile Representation (ICLR 2025 · similarity 0.91)
- Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper (NeurIPS 2025 · similarity 0.90)
- Binding Touch to Everything: Learning Unified Multimodal Tactile Representations (CVPR 2024 · similarity 0.90)
- Tactile MNIST: Benchmarking Active Tactile Perception (NeurIPS 2026 · similarity 0.90)
- Universal Visuo-Tactile Video Understanding for Embodied Interaction (NeurIPS 2025 · similarity 0.89)
- Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.