Research map / Video / Action recognition
Jointly Modeling Spatio-Temporal Features of Tactile Signals for Action Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STAT, a spatio-temporal aware tactility transformer with spatial and temporal embeddings plus a temporal pretraining task (tubelet time-order discrimination) for action classification from wearable sock tactile signals.
Problem
Action classification from tactile signals requires jointly modeling spatial (translation-variant sensor positions) and temporal features, which existing CNN-based and video/multivariate transformer methods fail to capture simultaneously.
Approach
A transformer over tactile tubelets with added sinusoidal spatial embeddings (sensor-region identity) and temporal embeddings (frame-block identity), pretrained with masked tubelet reconstruction using spatial-group masking plus a binary tubelet time-order discrimination task.
Key results
- STAT with pretraining reaches ACC@1 0.9033, ACC@3 0.9830, Macro-F1 0.9015, outperforming the best baseline CNN&GRU (0.8794 ACC@1, 0.8743 Macro-F1) by 2.7% ACC@1 and 3.1% Macro-F1.
- Ablation: temporal embeddings alone (0.8764 ACC@1) outperform spatial embeddings alone (0.8417); combining both with the temporal pretraining task gives the best ACC@1 and Macro-F1.
- VideoMAE performs worst (0.7705 ACC@1), and STAT without pretraining (0.8050 ACC@1) underperforms most baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add STAT's spatio-temporal modeling and pretraining | Tactile action classification · Tactile action classification test set | ACC@1: 0.8794 -> 0.9033 | Helped |
| add pretraining (ablation) | STAT · Tactile action classification test set | ACC@1: 0.8050 -> 0.9033 | Helped |
| add temporal embeddings (ablation) | STAT with spatial embeddings · Tactile action classification | ACC@1: 0.8417 -> 0.8947 | Helped |
| add spatial embeddings (ablation) | STAT with temporal embeddings · Tactile action classification | ACC@1: 0.8764 -> 0.8947 | Helped |
| add temporal embeddings (ablation) | STAT with spatial embeddings · Tactile action classification | Macro-F1: 0.8337 -> 0.8957 | Helped |
| add spatial embeddings (ablation) | STAT with temporal embeddings · Tactile action classification | Macro-F1: 0.8715 -> 0.8957 | Helped |
| add temporal pretraining (ablation) | STAT with both spatial and temporal embeddings · Tactile action classification | ACC@1: 0.8947 -> 0.9033 | Helped |
| add spatial and temporal embeddings (ablation) | STAT with temporal pretraining · Tactile action classification | ACC@1: 0.8299 -> 0.9033 | Helped |
| use N_comp=30 comparison pairs (ablation) | STAT temporal pretraining · Tactile action classification | classification performance | Helped |
| set temporal pretraining loss weight beta=1 rather than too low or too high (ablation) | STAT · Tactile action classification | classification performance | Helped |
Surprising findings
- VideoMAE, a video classification model, performs worst among baselines, suggesting direct reuse of video transformers is poor for tactile signals.
- STAT without pretraining performs worse than most baselines, so pretraining is necessary for the gains.
Limitations
- Evaluated on only one public tactile dataset, which the authors note is the only large-scale open tactile dataset available.
- Pretraining uses only the training split because no large open tactile datasets exist.
Experimental setup
Datasets: SensTextile tactile sock dataset
Benchmarks: SensTextile tactile sock dataset
Metrics: ACC@1 ACC@3 Macro-F1
Baselines: CNN&GRU TST VideoMAE STAT w/o pretraining
Builds on: VideoMAE TST Vision Transformer
Closest papers by meaning
- AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.86)
- Sensor-Invariant Tactile Representation (ICLR 2025 · similarity 0.85)
- Tactile MNIST: Benchmarking Active Tactile Perception (NeurIPS 2026 · similarity 0.85)
- Task-Optimized Convolutional Recurrent Networks Align with Tactile Processing in the Rodent Brain (NeurIPS 2025 · similarity 0.85)
- Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper (NeurIPS 2025 · similarity 0.84)
- EgoTac: In-the-wild Tactile Prediction from Egocentric Vision (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.