Research map / Video / Video segmentation and tracking

EgoTac: In-the-wild Tactile Prediction from Egocentric Vision

NeurIPS 2026 · Video › Video segmentation and tracking · approach: Physics-inspired motion prediction · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces EgoTac, an encoder-decoder model that predicts dense per-vertex MANO tactile force and contact from egocentric RGB clips, trained on a 5.7M-pair unified dataset including a new glove-based dataset (EgoTac-SC).

Problem

Tactile sensing is absent from most egocentric human manipulation datasets because it is hard to collect at scale; the paper asks whether dense tactile force and contact can be inferred purely from egocentric vision.

Approach

Map heterogeneous force-labeled and contact-only hand-object datasets onto a shared two-hand MANO topology and train a DINOv2-based temporal model with AdaLN tactile decoder using a joint force-contact objective, so contact-only data can supervise the contact head while force data supervises both heads.

Key results

• In-domain EgoTac achieves MAE_act of 0.056 N and temporal Pearson of 0.506 on the EgoTac-SC test set.
• On out-of-domain OAKINK2, EgoTac reaches AUROC 0.844 and F1 0.435, versus HACO at 0.772 and 0.400.
• On OpenTouch, EgoTac reaches force-cosine above 0.7 and rise/fall F1 above 0.4 after normalization, with scaling improving force-cosine to 0.787 at full data.

Surprising findings

• Removing explicit force supervision (contact-only training) raises global MAE to 0.322 N and drops temporal Pearson to 0.240, showing contact logits do not encode force scale.
• HACO achieves higher recall on OAKINK2 and FPHA, but the authors attribute this to over-predicting contact regions.

Limitations

• OOD force evaluation on OpenTouch is normalized to [0,1] because absolute Newton calibration is unavailable, so absolute force accuracy out-of-domain is not established.
• Contact-only datasets derive labels from mesh-proximity thresholds, which are analytical rather than physical contact.
• Zero-shot predictions on unconstrained videos are qualitative only, without quantitative ground truth.

Experimental setup

Datasets: EgoTac-SC HOI4D H2O HOT3D ARCTIC TACO OpenTouch OAKINK2 FPHA EgoDex EPIC-KITCHENS Ego4D

Benchmarks: EgoTac-SC held-out test HOT3D H2O ARCTIC OAKINK2 FPHA OpenTouch

Metrics: MAE MAE_act temporal Pearson AUROC IoU Precision Recall F1 force-cosine rise F1

Baselines: EgoTac-i BSTRO DECO HACO

Builds on: DINOv2 MANO AdamW

Closest papers by meaning

• Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper (NeurIPS 2025 · similarity 0.90)
• AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors (ICLR 2025 · similarity 0.88)
• PhysTacGen: Physics-Aware Visual-Tactile Sensor Image Generation (NeurIPS 2026 · similarity 0.88)
• HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing (NeurIPS 2026 · similarity 0.88)
• Tactile MNIST: Benchmarking Active Tactile Perception (NeurIPS 2026 · similarity 0.88)
• VinT-6D: A Large-Scale Object-in-hand Dataset from Vision, Touch and Proprioception (ICML 2024 · similarity 0.88)