Research map / Detection and segmentation / Open-set and few-shot detection
Toward Artificial Palpation: Representation Learning of Touch on Soft Bodies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes self-supervised encoder-decoder (FLE + GRU, force decoder) that learns a palpation representation from tactile sequences, then uses it for MRI-based tactile imaging and change detection on robot-collected breast phantoms and a simulator.
Problem
Artificial palpation: inferring the internal structure (e.g., lumps) of soft bodies from sequences of noisy, partial tactile force measurements, where ground-truth imaging data is scarce.
Approach
Pretrain an encoder-decoder on unlabeled tactile sequences by predicting forces at arbitrary past/future poses from a GRU sequence representation, then train a flow-matching image predictor on the frozen representation using limited MRI labels.
Key results
- Image prediction with pretrained representation reaches 74.4% F1 and 23.0% lump size error vs. 43.9% F1 with 0.25x data; MLP predictor reaches 10.5% size error.
- Change detection recall 0.82 and FAR 0.19 vs. human recall 0.62 and FAR 0.32 (63 samples, 16 participants).
- Adding unsupervised data alone raised F1 by over 11%, while adding supervised data on top contributed only about 2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the learned tactile representation instead of a force map for image prediction | Real-world tactile imaging · Real-world tactile imaging | F1 Score [%]: 47.3 ± 1.3 -> 74.4 ± 0.1 | Helped |
| Use an MLP predictor instead of image prediction for lump size | Pretrained tactile representation · Real-world tactile imaging | Size Error [%]: 23.0 ± 2.1 -> 10.5 ± 0.5 | Helped |
| Use 0.5× rather than full training data (ablation) | Image Pred. · Real-world tactile imaging | F1 Score [%]: 74.4 ± 0.1 -> 65.3 ± 0.1 | Hurt |
| Use 0.25× rather than full training data (ablation) | Image Pred. · Real-world tactile imaging | F1 Score [%]: 74.4 ± 0.1 -> 43.9 ± 2.6 | Hurt |
| Add full representation-learning data while keeping supervised data at 0.5× (ablation) | Tactile imaging · Real-world tactile imaging, 0.5× supervised data | F1 Score [%]: 65.3 ± 0.1 -> 72.7 ± 0.3 | Helped |
| Increase supervised data from 0.5× to 1× with full representation-learning data (ablation) | Tactile imaging · Real-world tactile imaging | F1 Score [%]: 72.7 ± 0.3 -> 74.4 ± 0.1 | Helped |
| Replace flow matching imaging with transposed convolutions (ablation) | GRU tactile imaging · Architecture ablation, 0.5× training data | Size Error [%]: 20.9 ± 0.9 -> 27.9 ± 1.8 | Hurt |
| Replace the GRU sequence encoder with a transformer (ablation) | Flow-matching tactile imaging · Architecture ablation, 0.5× training data | Size Error [%]: 20.9 ± 0.9 -> 35.4 ± 0.7 | Hurt |
| Use a masked transformer representation encoder (ablation) | Flow-matching tactile imaging · Architecture ablation, 0.5× training data | Size Error [%]: 20.9 ± 0.9 -> 57.9 ± 2.0 | Hurt |
| Mask input trajectories and reconstruct only the masked trajectories (ablation) | GRU representation encoder with flow matching · Architecture ablation, 0.5× training data | Size Error [%]: 20.9 ± 0.9 -> 23.9 ± 1.0 | Hurt |
| Use the learned method rather than human palpation | Change detection · Change detection | Recall: 0.62 -> 0.82 | Helped |
| Use the learned method rather than human palpation | Change detection · Change detection | FAR: 0.32 -> 0.19 | Helped |
Surprising findings
- Adding unsupervised tactile data alone gave most of the F1 gain, with labeled MRI data adding only about 2%.
- The Digit vision-based sensor performed poorly inside soft material, while the Xela uSkin worked.
Limitations
- Phantoms are not anatomically accurate; generalization to human breast tissue (ducts, vessels, contrast agents) is untested.
- Uses a single Xela uSkin sensor, fixed force and linear poking trajectories from a robot arm, requiring robot pose estimation.
- Predicts only a single horizontal MRI slice rather than full 3D volumes; small human study (16 participants, 63 samples).
Experimental setup
Datasets: Robot-collected tactile palpation data from breast phantoms (~60K pokes) MRI scans of phantom inserts (~200 bodies) PalpationSim simulated palpation data
Benchmarks: PalpationSim simulated imaging Real-world breast phantom MRI tactile imaging Human change detection study
Metrics: F1 score lump size error center-of-mass error recall false alarm rate
Baselines: Supervised image prediction (no pretraining) Force Map + Image Pred. Average Pred. Human palpation (change detection) TR + FM GRU + TC
Builds on: flow matching BYOL masked autoencoders panda-py
Closest papers by meaning
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.87)
- EgoTac: In-the-wild Tactile Prediction from Egocentric Vision (NeurIPS 2026 · similarity 0.87)
- Task-Optimized Convolutional Recurrent Networks Align with Tactile Processing in the Rodent Brain (NeurIPS 2025 · similarity 0.85)
- Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper (NeurIPS 2025 · similarity 0.85)
- Sensor-Invariant Tactile Representation (ICLR 2025 · similarity 0.85)
- Binding Touch to Everything: Learning Unified Multimodal Tactile Representations (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.