Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Sensor-Invariant Tactile Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SITR, a calibration-conditioned ViT pre-trained on 1M simulated GelSight-style tactile images (100 sensor configs) with normal-map reconstruction and supervised contrastive losses, enabling zero-shot transfer of frozen tactile features across real optical tactile sensors.
Problem
Vision-based tactile sensors differ in optical design and manufacturing, so models trained on one sensor fail to transfer to others; collecting per-sensor data is costly.
Approach
Pre-train a ViT-Base encoder that takes both the tactile image and a set of sensor-specific calibration images (ball and cube presses) as tokens, supervised by normal-map reconstruction and supervised contrastive learning across many simulated sensor configurations, then freeze it and train light decoders for downstream tasks.
Key results
- SITR achieves 81.94% inter-sensor transfer classification accuracy vs. 54.34% for ViT-Large pre-trained and 48.10% for ViT-Base pre-trained.
- On inter-sensor pose estimation SITR reaches 0.80 mm RMSE, versus 1.49 mm for the best baseline (ViT-Large), roughly a 50% reduction.
- Ablation: normal + SCL losses give 91.43% classification vs. 84.21% (normal only) and 78.86% (SCL only); transfer accuracy grows with both sensor variations and samples per sensor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SITR instead of ViT-Large Pre-trained | object classification · Inter-sensor set | accuracy (%): 54.34 ± 23.04 -> 81.94 ± 12.92 | Helped |
| Use SITR instead of ViT-Large Pre-trained | object classification · Intra-sensor set | accuracy (%): 78.38 ± 17.79 -> 90.23 ± 8.16 | Helped |
| Use SITR instead of ViT-Large Pre-trained | object classification · Wedge-Mini | accuracy (%): 79.04 ± 16.44 -> 90.80 ± 2.85 | Helped |
| Use SITR instead of ViT-Large Pre-trained | pose estimation · Inter-sensor set | RMSE (mm): 1.49 ± 0.25 -> 0.80 ± 0.21 | Helped |
| Use SITR instead of ViT-Large Pre-trained | pose estimation · Wedge-Mini | RMSE (mm): 1.45 ± 0.01 -> 0.62 ± 0.11 | Helped |
| Add normal map reconstruction loss to SCL loss (ablation) | SITR · Classification transfer | accuracy (%): 78.86 ± 18.72 -> 91.43 ± 9.88 | Helped |
| Add SCL loss to normal map reconstruction loss (ablation) | SITR · Classification transfer | accuracy (%): 84.21 ± 14.01 -> 91.43 ± 9.88 | Helped |
| Replace normal map supervision with MAE reconstruction (ablation) | SITR · Classification, inter-sensor set | accuracy (%): 81.94 ± 12.92 -> 26.46 ± 19.54 | Hurt |
| Replace normal map supervision with VQGAN reconstruction (ablation) | SITR · Classification, inter-sensor set | accuracy (%): 81.94 ± 12.92 -> 31.02 ± 22.01 | Hurt |
| Replace normal map supervision with MAE reconstruction (ablation) | SITR · Pose estimation, inter-sensor set | RMSE (mm): 0.80 ± 0.21 -> 1.13 ± 0.19 | Hurt |
| Replace normal map supervision with VQGAN reconstruction (ablation) | SITR · Pose estimation, inter-sensor set | RMSE (mm): 0.80 ± 0.21 -> 1.18 ± 0.14 | Hurt |
| Increase simulated samples per sensor from 1K to 10K (ablation) | SITR · Classification, inter-sensor set; 100 sensor variations | accuracy (%): 62.85 ± 16.45 -> 81.94 ± 12.92 | Helped |
Surprising findings
- Most baselines perform well in the no-transfer setting but collapse when tested on a different sensor, indicating sensor-specific features.
- Pose estimation shows ImageNet pre-training gives only marginal gains over training from scratch for tactile regression.
Limitations
- Trained and evaluated mainly on flat-gel GelSight-family sensors; DIGIT with distinct optics transfers worse.
- Evaluated only on classification and pose estimation with a limited set of real sensors (7) and objects (16).
Experimental setup
Datasets: simulated tactile dataset (1M images, Blender PBR) real-world classification dataset (112K images, 16 objects, 7 sensors) real-world pose estimation dataset (24K images, 4 sensors)
Benchmarks: real-world object classification (16 classes) real-world pose estimation (3-DoF RMSE)
Metrics: top-1 accuracy RMSE (mm)
Baselines: ViT-Base scratch ViT-Base ImageNet pre-trained ViT-Large pre-trained T3 UniT MAE VQGAN
Builds on: Vision Transformer supervised contrastive learning Masked Auto-Encoder
Closest papers by meaning
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.91)
- Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper (NeurIPS 2025 · similarity 0.89)
- Tactile MNIST: Benchmarking Active Tactile Perception (NeurIPS 2026 · similarity 0.88)
- Universal Visuo-Tactile Video Understanding for Embodied Interaction (NeurIPS 2025 · similarity 0.88)
- Touch-R1: Reinforcing Touch Reasoning in MLLMs (NeurIPS 2026 · similarity 0.87)
- DIFFTACTILE: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.