Research map / Probabilistic and scientific modeling / Dynamical systems
DIFFTACTILE: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DiffTactile, a differentiable, FEM-based tactile simulator (with MPM/PBD multi-material objects and a penalty contact model) used for sim-to-real system identification, learned optical rendering, and gradient-based tactile-assisted manipulation learning.
Problem
Existing tactile simulators mostly handle rigid bodies or use simplified soft-contact approximations, are not differentiable, and do not support diverse deformable materials, which limits sim-to-real transfer and efficient learning of contact-rich tactile-assisted manipulation.
Approach
A system-wide differentiable simulator combining FEM for the soft sensor elastomer, MLS-MPM for rigid/elastic/elastoplastic objects, PBD for cables, and a penalty-based contact model, coupled with an MLP-based pixel optical renderer; gradients through the simulator drive both system identification and trajectory optimization.
Key results
- System identification on Real2Sim: Ours achieves marker MSE 3.08 on press-slide vs. 3.42 for CMA-ES and 3.54 for Random.
- Optical simulation: the MLP renderer reduces L1 from 16.1 (Taxim) to 7.94 and MSE from 85.74 to 56.10 on the held-out set.
- Manipulation: gradient-based optimization achieves ObjectRepose 60.82 (w/ tactile) vs. 5.00 for SAC and CaseOpen 17.11 w/o tactile vs. -0.68 for SAC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use gradient-based trajectory optimization instead of the strongest baseline | DiffTactile system identification · Sim2Sim press-slide, press-twist-z, and press-twist-x | marker MSE: 0.59± 0.12 -> 0.53± 0.35; 0.47± 0.15 -> 0.42± 0.24; 0.61± 0.20 -> 0.58± 0.28 | Helped |
| Use gradient-based trajectory optimization instead of the strongest baseline | DiffTactile system identification · Real2Sim press-slide, press-twist-z, and press-twist-x | marker MSE: 3.29± 1.51 -> 3.08± 1.27; 2.42± 0.90 -> 2.38± 0.86; 4.47± 3.31 -> 3.99± 2.89 | Helped |
| Use the pixel-based neural optical model instead of Taxim's polynomial mapping | DiffTactile optical simulation · Optical simulation test set | L1, MSE, SSIM, and PSNR: 16.1 -> 7.94 L1; 85.74 -> 56.10 MSE; 0.998 -> 0.999 SSIM; 38.47 -> 39.42 PSNR | Helped |
| Add tactile feedback to grasp control | Elastic and elastoplastic grasping · Elastic grasp | position loss and slipping distance: 0.04± 0.03 -> 0.01± 0.01 position loss; 0.18± 0.06 -> 0.07± 0.04 slipping distance | Helped |
| Add tactile feedback to grasp control | Elastic and elastoplastic grasping · Elastoplastic grasp | position loss and slipping distance: 0.70± 0.53 -> 0.24± 0.01 position loss; 0.26± 0.09 -> 0.2± 0.05 slipping distance | Helped |
| Add tactile feedback to grasp control | Elastoplastic grasping · Elastoplastic grasp | deformation loss: 0.85± 0.25 -> 0.89± 0.48 | Hurt |
| Use gradient-based optimization instead of the strongest listed baseline | DiffTactile contact-rich manipulation · Object reposing | orientation change: 5.00± 0.01 -> 60.82± 0.00 | Helped |
| Use gradient-based optimization instead of the strongest listed baseline | DiffTactile contact-rich manipulation · Cable straightening | aggregation distance: 1.36± 0.03 -> 0.89± 0.32 | Helped |
| Use gradient-based optimization instead of the strongest listed baseline | DiffTactile contact-rich manipulation · Case opening | orientation change: -0.68± 0.29 -> 17.11± 0.05 | Helped |
| Use gradient-based optimization instead of the strongest listed baseline | DiffTactile contact-rich manipulation · Surface following | in-contact travel distance: 14.00± 2.00 -> 51.67± 12.86 | Helped |
| Add tactile reward to gradient-based optimization | DiffTactile contact-rich manipulation · Object reposing | orientation change: 12.07± 12.46 -> 60.82± 0.00 | Helped |
| Add tactile reward to gradient-based optimization | DiffTactile contact-rich manipulation · Cable straightening | aggregation distance: 1.27± 0.81 -> 0.89± 0.32 | Helped |
Surprising findings
- Adding tactile observations did not uniformly help RL baselines; PPO and SAC losses did not decrease within 100 iterations, which the authors attribute partly to insufficient data and reward design.
- For CaseOpen, the gradient method without tactile reward (17.11) outperformed the variant with tactile reward (9.83).
Limitations
- Optimized contact and object parameters may not transfer across object materials; friction and object parameters are manually randomized.
- RL baselines were not tuned extensively; authors note simple reward designs and limited iterations.
- Sim-to-real validation is limited to a few real-world tasks (surface following, case opening, deformable grasp) with a single GelSight-style sensor.
Experimental setup
Datasets: EGAD object dataset real-world GelSight press-slide/twist sequences 250 real indentation examples collected with 4mm sphere
Benchmarks: DiffTactile system identification sequences (press-slide, press-twist-z, press-twist-x) optical simulation test set grasping tasks (elastic, elastoplastic) ObjectRepose, CableStraighten, CaseOpen, SurfaceFollow
Metrics: pixel-wise MSE L1 SSIM PSNR position loss slip distance deformation loss task-specific rewards/distances
Baselines: Random RNN CMA-ES SAC PPO Taxim
Builds on: Taichi MLS-MPM Position-Based Dynamics Taxim DiffRedMax
Closest papers by meaning
- Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation (NeurIPS 2025 · similarity 0.90)
- Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper (NeurIPS 2025 · similarity 0.88)
- Tactile MNIST: Benchmarking Active Tactile Perception (NeurIPS 2026 · similarity 0.87)
- Sensor-Invariant Tactile Representation (ICLR 2025 · similarity 0.87)
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.87)
- PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.