Research map / Vision-language and multimodal / Human and embodied modeling
TextToucher: Fine-Grained Text-to-Touch Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TextToucher, a text-to-touch diffusion transformer that conditions tactile image generation on MLLM-generated object-level texture/shape text and learnable gel-status prompts, plus a CTTP contrastive metric for text-tactile alignment.
Problem
Generating tactile images from text descriptions rather than visual images, since existing vision-to-touch methods cannot accurately capture human tactile descriptions and ignore gel status differences across sensors.
Approach
Decompose tactile conditioning into object-level text (texture from dataset annotations, shape captioned by LLaVA from vision images) and sensor-level learnable gel-status prompts, fuse them in a DiT with cross-attention and time-adaptive conditioning, and evaluate alignment with a contrastively trained tactile-text encoder (CTTP).
Key results
- On HCT, TextToucher reaches CTTP 0.261, LPIPS 0.427, SSIM 0.904, PSNR 22.70, beating PixArt-α (CTTP 0.198, LPIPS 0.504) and UniTouch/GVST.
- On SSVTP, TextToucher reaches CTTP 0.152 and LPIPS 0.465, versus 0.125 and 0.497 for PixArt-α; PixArt-α has slightly higher PSNR (22.35 vs 22.43 gap is small in TextToucher's favor).
- Cross attention conditioning outperforms text modulation (CTTP 0.151) and joint attention (CTTP 0.213) on HCT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add tactile shape descriptions to texture-only conditions (ablation) | TextToucher · HCT | CTTP: 0.198 -> 0.236 | Helped |
| Add gel status to texture and shape conditions (ablation) | TextToucher · HCT | CTTP: 0.236 -> 0.261 | Helped |
| Use cross attention instead of text modulation or joint attention (ablation) | TextToucher · HCT | CTTP: 0.151; 0.213 -> 0.261 | Helped |
| Apply gel status prompts in shallow layers, extending them through layer 28 (ablation) | TextToucher · HCT | CTTP: 0.258 -> 0.261 | Helped |
| Apply gel status prompts in deeper-to-shallower layers (14-28 or 7-28) (ablation) | TextToucher · HCT | CTTP: 0.241; 0.249 | Hurt |
| Set gel-status prompt token count to 4 rather than 1, 2, 6, or 8 (ablation) | TextToucher · HCT | CTTP: 0.248; 0.255; 0.261; 0.250; 0.252 | Helped |
| Use a timestep threshold of 600 or 800 for switching text conditions (ablation) | TextToucher · HCT | CTTP: 0.258; 0.261; 0.247; 0.235; 0.234 | Helped |
| Use TextToucher instead of PixArt-α | Text-to-touch generation · HCT | CTTP: +0.063 | Helped |
Surprising findings
- Traditional text-to-image method PixArt-α performs comparably on SSVTP, suggesting general text-to-image models are inadequate for tactile generation.
- Adding gel status prompts only in deeper layers (14-28) or from deep to shallow degrades performance versus applying from shallow layers.
Limitations
- Gel state 1 has few samples (2.4% of HCT), leading to poor generation quality for that status.
- Object-level shape is approximated by the contacted object's caption from vision images rather than true tactile shape.
Experimental setup
Datasets: HCT SSVTP
Benchmarks: HCT SSVTP
Metrics: CTTP LPIPS SSIM PSNR
Baselines: GVST UniTouch PixArt-α
Builds on: DiT PixArt-α CLIP LLaVA DPM-Solver
Closest papers by meaning
- A Touch, Vision, and Language Dataset for Multimodal Alignment (ICML 2024 · similarity 0.89)
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.88)
- PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generation (NeurIPS 2026 · similarity 0.87)
- Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.87)
- Tactile-Augmented Radiance Fields (CVPR 2024 · similarity 0.87)
- Universal Visuo-Tactile Video Understanding for Embodied Interaction (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.