Research map / Vision-language and multimodal / Human and embodied modeling
Touch and Go: Learning from Human-Collected Vision and Touch
Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, Andrew Owens
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visual input to tactile history for future touch prediction (Geng et al.) (ablation) | Geng et al. video prediction · future touch prediction, skip 5 frames | L1: 0.691 -> 0.663 | Improved |
| add visual input to tactile history (Geng et al.) (ablation) | Geng et al. video prediction · future touch prediction, skip 3 frames | LPIPS: 0.103 -> 0.091 | Improved |
| add visual input to tactile history (SVG) (ablation) | SVG video prediction · future touch prediction, skip 5 frames | L1: 0.807 -> 0.762 | Improved |
| increase fraction of material classes in pretraining from 25% to 100% (ablation) | visuo-tactile CMC · Touch and Go material classification | accuracy (%): 25.7 -> 54.7 | Improved |
| increase fraction of material classes in pretraining from 25% to 100% (ablation) | visuo-tactile CMC · robot grasping | accuracy (%): 62.3 -> 78.1 | Improved |
| pretrain visuo-tactile CMC on Touch and Go instead of supervised ImageNet features | tactile representation · robot grasping | accuracy (%): 73.0 -> 78.1 | Improved |
| pretrain on Touch and Go instead of Object Folder 2.0 | visuo-tactile CMC · Touch and Go material classification | accuracy (%): 36.2 -> 54.7 | Improved |
| use CUT-based stylization instead of CycleGAN (ablation) | tactile-driven image stylization · tactile-driven stylization | CMC similarity: 0.178 -> 0.197 | Improved |
Closest papers by meaning
- EgoTac: In-the-wild Tactile Prediction from Egocentric Vision (NeurIPS 2026 · similarity 0.92)
- Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper (NeurIPS 2025 · similarity 0.90)
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.89)
- Binding Touch to Everything: Learning Unified Multimodal Tactile Representations (CVPR 2024 · similarity 0.88)
- Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.88)
- Universal Visuo-Tactile Video Understanding for Embodied Interaction (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
