Research map / Vision-language and multimodal / Robot action and navigation
Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents a portable handheld visuo-tactile gripper and dataset (2.6M pairs, 2,700+ demos), plus a masked tactile-reconstruction visuo-tactile encoder with cross-attention feeding a diffusion policy for fine-grained manipulation.
Problem
Handheld demonstration grippers lack tactile sensing and large-scale in-the-wild visuo-tactile data, and existing visuo-tactile fusion is poorly suited to learning from unstructured multimodal data for fine-grained manipulation.
Approach
Pretrain a CLIP-initialized ViT and tactile CNN jointly with a masked tactile reconstruction objective and two-round cross-attention fusion, so that modality-specific structure is preserved, then condition a diffusion policy on the fused embedding.
Key results
- On Test Tube Collection, the full method reaches 0.85 whole-task success vs 0.25 for vision-only and 0.70 without pretraining.
- On Fluid Transfer, the full method reaches 0.90 whole-task success vs 0.55 for vision-only and 0.80 without pretraining.
- On Whiteboard Erasing, the full method reaches 0.70 vs 0.55 vision-only and 0.45 without cross-attention; Pencil Insertion reaches 0.85 vs 0.45 vision-only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visuo-tactile pretraining (ablation) | visuo-tactile policy · Test Tube Collection | whole-task success rate: 0.70 -> 0.85 | Helped |
| add visuo-tactile pretraining (ablation) | visuo-tactile policy · Pencil Insertion | whole-task success rate: 0.75 -> 0.85 | Helped |
| add visuo-tactile pretraining (ablation) | visuo-tactile policy · Fluid Transfer | whole-task success rate: 0.80 -> 0.90 | Helped |
| add visuo-tactile pretraining (ablation) | visuo-tactile policy · Whiteboard Erasing | whole-task success rate: 0.60 -> 0.70 | Helped |
| add cross-attention rather than concatenating modality features (ablation) | visuo-tactile policy · Test Tube Collection | whole-task success rate: 0.50 -> 0.85 | Helped |
| add cross-attention rather than concatenating modality features (ablation) | visuo-tactile policy · Pencil Insertion | whole-task success rate: 0.70 -> 0.85 | Helped |
| add cross-attention rather than concatenating modality features (ablation) | visuo-tactile policy · Fluid Transfer | whole-task success rate: 0.70 -> 0.90 | Helped |
| add cross-attention rather than concatenating modality features (ablation) | visuo-tactile policy · Whiteboard Erasing | whole-task success rate: 0.45 -> 0.70 | Helped |
| add tactile sensing and visuo-tactile input | policy compared with Vision-Only · Test Tube Collection | whole-task success rate: 0.25 -> 0.85 | Helped |
| add tactile sensing and visuo-tactile input | policy compared with Vision-Only · Pencil Insertion | whole-task success rate: 0.45 -> 0.85 | Helped |
| add tactile sensing and visuo-tactile input | policy compared with Vision-Only · Fluid Transfer | whole-task success rate: 0.55 -> 0.90 | Helped |
| add tactile sensing and visuo-tactile input | policy compared with Vision-Only · Whiteboard Erasing | whole-task success rate: 0.55 -> 0.70 | Helped |
Surprising findings
- Removing all task-specific demonstrations from pretraining leaves Test Tube Collection performance nearly unchanged (0.80 vs 0.85), and Whiteboard Erasing even improves (0.75 vs 0.70).
- Policies without cross-attention over-rely on tactile signals, applying excessive force that triggers safety stops during whiteboard erasing.
Limitations
- Evaluation limited to four tasks with 20 trials each on a single-arm parallel-gripper setup.
- Attention and reconstruction results are qualitative; the paper relies on visual inspection of attention maps.
- Authors note the approach has not yet been extended to multi-finger dexterous hands.
Experimental setup
Datasets: in-the-wild visuo-tactile dataset (2.6M pairs, 2,700+ demos, 43 tasks)
Benchmarks: Test Tube Collection Pencil Insertion Fluid Transfer Whiteboard Erasing
Metrics: success rate over 20 trials per task per-subtask success rate test MSE reconstruction loss
Baselines: Vision-Only (Diffusion Policy with CLIP) Ours w/o Cross-Attention Ours w/o Pretraining
Builds on: Diffusion Policy CLIP 3D-ViTac masked autoencoder
Closest papers by meaning
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.90)
- EgoTac: In-the-wild Tactile Prediction from Egocentric Vision (NeurIPS 2026 · similarity 0.90)
- Enhancing Tactile-based Reinforcement Learning for Robotic Control (NeurIPS 2025 · similarity 0.90)
- Tactile MNIST: Benchmarking Active Tactile Perception (NeurIPS 2026 · similarity 0.89)
- Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.89)
- Sensor-Invariant Tactile Representation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.