Research map / 3D vision / NeRF and radiance fields
Tactile-Augmented Radiance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a tactile-augmented radiance field by registering vision-based touch sensor images to a NeRF and training a conditional latent diffusion model to impute touch at novel scene locations, with a new scene-scale visual-tactile dataset of 19.3k pairs.
Problem
Touch signals are expensive to collect and existing visual-tactile datasets lack spatially aligned vision and touch at scene scale, so touch cannot be estimated for unprobed locations in a scene.
Approach
Register touch-sensor images to NeRF-rendered views via multi-view geometry (sensor is a perspective camera), then condition a latent diffusion model on NeRF RGB-D renders to propagate sparse touch probes to visually and structurally similar regions.
Key results
- Diffusion model achieves FID 28.97 versus 130.22 for VisGel and CVTP 0.80 versus 0.03, though PSNR/SSIM are comparable or slightly lower.
- 3D tactile localization mAP at r=0.1m improves from 57.15 (real data) to 60.61 when adding estimated touch samples.
- Material classification accuracy improves from 54.7 (Touch and Go) to 59.0 when pretraining also uses real plus estimated samples from the proposed data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the TaRF diffusion model instead of VisGel | Touch estimation model · Touch estimation (novel views) test set | FID: 130.22 -> 28.97 | Helped |
| Train the localization model with estimated touch samples in addition to real samples | 3D tactile localization model · 3D tactile localization, r=0.001–0.1 m | mAP: 12.10 -> 14.92; 22.93 -> 26.69; 32.10 -> 36.17; 50.30 -> 53.62; 57.15 -> 60.61 | Helped |
| Add estimated tactile probes to Touch and Go pretraining data | Material classification model · Material classification | accuracy: 57.6 -> 59.0 | Helped |
| Add estimated tactile probes to Touch and Go pretraining data | Material classification model · Hard/soft classification | accuracy: 88.4 -> 88.7 | Helped |
| Add estimated tactile probes to Touch and Go pretraining data | Material classification model · Rough/smooth classification | accuracy: 81.7 -> 86.1 | Helped |
| Add RGB conditioning (ablation) | TaRF diffusion model · Touch estimation (novel views) test set | FID: 34.31 -> 28.97 | Helped |
| Add depth conditioning (ablation) | TaRF diffusion model · Touch estimation (novel views) test set | FID: 33.16 -> 28.97 | Helped |
| Add contrastive pretraining (ablation) | TaRF diffusion model · Touch estimation (novel views) test set | FID: 32.98 -> 28.97 | Helped |
| Use contrastive re-ranking (ablation) | TaRF diffusion model · Touch estimation (novel views) test set | CVTP: 0.61 -> 0.80 | Helped |
| Use multiscale conditioning (ablation) | TaRF diffusion model · Touch estimation (novel views) test set | FID: 30.89 -> 28.97 | Helped |
Surprising findings
- The L1 regression baseline achieves the best PSNR and SSIM, showing these low-level metrics do not capture tactile realism.
Limitations
- Centimeter-scale SfM or registration errors cause several-pixel misalignments between NeRF views and touch samples.
- Assumes the scene's coarse structure does not deform when touched, which may fail for inelastic surfaces.
- Generated touch images can be geometrically misaligned with ground truth in failure cases.
Experimental setup
Datasets: YCB-Slide Touch and Go ObjectFolder 2.0 VisGel
Benchmarks: TaRF dataset test split
Metrics: FID PSNR SSIM CVTP mAP classification accuracy
Baselines: VisGel L1 regression model Touching a NeRF ObjectFolder 2.0 Touch and Go
Builds on: Stable Diffusion Nerfacto CLIP-style InfoNCE contrastive learning VisGel SSVTP
Closest papers by meaning
- Tactile DreamFusion: Exploiting Tactile Sensing for 3D Generation (NeurIPS 2024 · similarity 0.89)
- PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generation (NeurIPS 2026 · similarity 0.88)
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.87)
- TextToucher: Fine-Grained Text-to-Touch Generation (AAAI 2025 · similarity 0.87)
- Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction (CVPR 2025 · similarity 0.86)
- Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.