Research map / Vision-language and multimodal / Human and embodied modeling
Connecting Touch and Vision via Cross-Modal Prediction
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add reference visual and tactile images to the conditional GAN | Cross-modal prediction model · VisGel, vision↔touch | Prediction realism and input-output alignment | Helped |
| Add data rebalancing (ablation) | Vision→touch model · VisGel, vision→touch, seen objects | Turkers labeled real (%): 39.95% -> 46.63% | Helped |
| Add data rebalancing (ablation) | Vision→touch model · VisGel, vision→touch, unseen objects | Turkers labeled real (%): 34.86% -> 38.22% | Helped |
| Add temporal input frames (ablation) | Vision→touch model · VisGel, vision→touch, seen objects | Turkers labeled real (%): 41.44% -> 46.63% | Helped |
| Add temporal input frames (ablation) | Vision→touch model · VisGel, vision→touch, unseen objects | Turkers labeled real (%): 31.60% -> 38.22% | Helped |
| Add temporal input frames (ablation) | Touch→vision model · VisGel, touch→vision, seen objects | Turkers labeled feels similar (%): 81.31% -> 89.20% | Helped |
| Add temporal input frames (ablation) | Touch→vision model · VisGel, touch→vision, unseen objects | Turkers labeled feels similar (%): 78.40% -> 83.44% | Helped |
| Add temporal input frames (ablation) | Touch→vision model · VisGel, touch→vision, seen objects | Turkers labeled real (%): 30.80% -> 30.50% | No clear effect |
| Add temporal input frames (ablation) | Touch→vision model · VisGel, touch→vision, unseen objects | Turkers labeled real (%): 20.74% -> 24.22% | Helped |
| Use the full self-supervised model instead of pix2pix | Touch→vision model · VisGel, touch→vision, seen objects | Turkers labeled feels similar (%): 44.52% -> 89.20% | Helped |
| Use the full self-supervised model instead of pix2pix | Touch→vision model · VisGel, touch→vision, unseen objects | Turkers labeled feels similar (%): 25.21% -> 83.44% | Helped |
| Use the full self-supervised model instead of supervised prediction | Touch→vision model · VisGel, touch→vision, seen objects | Turkers labeled feels similar (%): 90.37% -> 89.20% | Hurt |
Closest papers by meaning
- Tactile-Augmented Radiance Fields (CVPR 2024 · similarity 0.90)
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.90)
- EgoTac: In-the-wild Tactile Prediction from Egocentric Vision (NeurIPS 2026 · similarity 0.90)
- Universal Visuo-Tactile Video Understanding for Embodied Interaction (NeurIPS 2025 · similarity 0.90)
- Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.89)
- Binding Touch to Everything: Learning Unified Multimodal Tactile Representations (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.