Research map / 3D vision / Face and shape
Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Touch2Shape uses a touch-conditioned 3D latent diffusion model (VQ-VAE T-SDF latent space) for tactile-only or visual-tactile 3D shape reconstruction, a touch shape fusion refinement module, and a DQN-trained touch exploration policy guided by diffusion-loss rewards.
Problem
Reconstructing complex 3D shapes from tactile (and optional visual) observations is hard because vision suffers from occlusion and lighting and existing 3D diffusion models rely on predetermined partial observations rather than active touch exploration.
Approach
Condition a latent 3D diffusion model on touch-chart tokens, align touch and shape latents with a MoCo-style contrastive encoder, fuse voxelized touch history into the VQ-VAE decoder, and use the diffusion model's denoising-loss change as the reward for training a touch-location policy, avoiding full shape decoding at each step.
Key results
- On ABC with 1 grasp in tactile-only mode, the method reaches CD 40.283 in the table's Grasp 1 column, against 24.864 for ActiveVT; the table is ambiguous here, and the 6.794 value in the Grasp 1 column may be the more relevant one.
- On ABC visual-tactile with one grasp, CD drops to 1.475 versus 2.538 for ActiveVT.
- On ShapeNet (EMD, 20 touches, tactile+visual), average EMD is 0.042 versus 0.081 for TouchSDF; tactile-only starts worse than TouchSDF at 1 touch (0.124 vs 0.136) but is better at 10 touches (0.056 vs 0.112).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Touch2Shape instead of ActiveVT | ABC tactile-only reconstruction · ABC, 1 grasp | CD: 8.220 -> 6.794 | Helped |
| Use Touch2Shape instead of ActiveVT | ABC visual-tactile reconstruction · ABC, 1 grasp | CD: 2.486 -> 1.406 | Helped |
| Add visual input to tactile-only reconstruction | Touch2Shape · ShapeNet, 1 touch, average | EMD: 0.124 -> 0.048 | Helped |
| Increase the number of touches from 1 to 20 | Touch2Shape, tactile only · ShapeNet, average | EMD: 0.124 -> 0.053 | Helped |
| Use the learned RL exploration policy instead of random actions | Touch2Shape, tactile-only exploration · ABC, 5 actions | CD ratio (%): 8.14 -> 6.63 | Helped |
| Use the learned RL exploration policy instead of random actions | Touch2Shape, visual-tactile exploration · ABC, 5 actions | CD ratio (%): 88.41 -> 86.96 | Helped |
| Add the contrastive encoder and touch shape fusion module (ablation) | Touch2Shape, tactile-only reconstruction · ABC, 5 random grasps | CD: 4.430 -> 3.134 | Helped |
| Add the contrastive encoder (ablation) | Touch2Shape, tactile-only reconstruction · ABC, 5 random grasps | CD: 3.298 -> 3.134 | Helped |
| Add the contrastive encoder (ablation) | Touch2Shape, visual-only reconstruction · ABC, 5 random grasps | CD: 2.242 -> 2.068 | Helped |
| Add touch shape fusion (ablation) | Touch2Shape, tactile-only reconstruction · ABC, 5 random grasps | CD: 3.298 -> 3.134 | Helped |
Surprising findings
- Tactile-only Touch2Shape is worse than TouchSDF with a single touch on ShapeNet (EMD 0.124 vs 0.136 is actually slightly better; the paper states its method was initially not ideal, so the comparison is mixed by category).
Limitations
- Experiments are only in simulation; transfer to a real robot is future work.
- Initial performance is weaker than TouchSDF on ShapeNet when few touches are available.
- Reward design and Table 1 results contain inconsistencies between column values, making some numbers ambiguous.
Experimental setup
Datasets: ABC ShapeNet
Benchmarks: ABC (CD) ShapeNet six categories (EMD)
Metrics: Chamfer Distance Earth Mover's Distance
Baselines: VTRecon ActiveVT TouchSDF
Builds on: SDFusion VQ-VAE ActiveVT VTRecon MoCo
Closest papers by meaning
- Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation (NeurIPS 2024 · similarity 0.88)
- Tactile-Augmented Radiance Fields (CVPR 2024 · similarity 0.86)
- PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generation (NeurIPS 2026 · similarity 0.84)
- TextToucher: Fine-Grained Text-to-Touch Generation (AAAI 2025 · similarity 0.84)
- EgoTac: In-the-wild Tactile Prediction from Egocentric Vision (NeurIPS 2026 · similarity 0.84)
- Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.