Research map / 3D vision / 3D generation
Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds high-resolution GelSight tactile normal maps as a second modality to 3D generation, co-optimizing albedo and tactile normal textures in a 3D texture field guided by ControlNet and a tactile TextureDreambooth, plus a diffusion-based multi-part texturing pipeline.
Problem
Text- and image-to-3D generation methods produce overly smooth surfaces or bake fine geometric details into albedo maps, because high-resolution geometric texture data is absent from 2D/3D datasets and hard to describe in text.
Approach
Capture fine surface geometry with a GelSight tactile sensor, convert it to normal patches, and jointly optimize a 3D texture field of albedo and tactile normals with visual (normal-conditioned ControlNet) and tactile (LoRA-customized DreamBooth) diffusion guidance, so visual and geometric textures stay aligned.
Key results
- Human study on AMTurk (1,000 responses): ours preferred over DreamGaussian 85.43% (texture) and 92.85% (geometry).
- Ours vs Wonder3D: 86.36% texture and 88.07% geometry preference.
- Ours vs DreamCraft3D: 61.54% texture and 84.20% geometry preference; method reportedly needs about 10x less compute than DreamCraft3D.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Tactile DreamFusion instead of DreamGaussian | 3D generation · AMTurk texture appearance | preference score: 85.43 \pm 4.76 | Helped |
| Use Tactile DreamFusion instead of DreamGaussian | 3D generation · AMTurk geometric details | preference score: 92.85 \pm 3.47 | Helped |
| Use Tactile DreamFusion instead of Wonder3D | 3D generation · AMTurk texture appearance | preference score: 86.36 \pm 3.93 | Helped |
| Use Tactile DreamFusion instead of Wonder3D | 3D generation · AMTurk geometric details | preference score: 88.07 \pm 3.80 | Helped |
| Use Tactile DreamFusion instead of DreamCraft3D | 3D generation · AMTurk texture appearance | preference score: 61.54 \pm 4.43 | Helped |
| Use Tactile DreamFusion instead of DreamCraft3D | 3D generation · AMTurk geometric details | preference score: 84.20 \pm 4.17 | Helped |
| Add high-pass filtering and contact-area cropping to tactile data preprocessing (ablation) | Tactile DreamFusion · 3D texture generation | geometric detail realism | Helped |
| Add tactile input while retaining the refinement loss (ablation) | Tactile DreamFusion · 3D texture generation | fine-grained geometric details | Helped |
| Add tactile guidance loss (ablation) | Tactile DreamFusion · 3D texture refinement | tactile texture details | Helped |
| Add visual guidance loss (ablation) | Tactile DreamFusion · 3D texture refinement | visual-tactile alignment | Helped |
Limitations
- Coarse shape quality depends on the underlying 3D generative model, which struggles with complex geometry.
- Slight seams can appear due to UV unwrapping.
- Requires physically pressing a GelSight sensor on a real object to collect tactile patches.
Experimental setup
Datasets: TouchTexture
Benchmarks: TouchTexture
Metrics: human preference rate (%) on texture appearance human preference rate (%) on geometric details
Baselines: DreamCraft3D DreamGaussian Wonder3D InstantMesh RichDreamer
Builds on: DreamFusion DreamBooth Wonder3D ControlNet Image Quilting
Closest papers by meaning
- PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generation (NeurIPS 2026 · similarity 0.89)
- Tactile-Augmented Radiance Fields (CVPR 2024 · similarity 0.89)
- Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction (CVPR 2025 · similarity 0.88)
- TextToucher: Fine-Grained Text-to-Touch Generation (AAAI 2025 · similarity 0.86)
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.85)
- Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.