Research map / Vision-language and multimodal / Robot action and navigation
VinT-6D: A Large-Scale Object-in-hand Dataset from Vision, Touch and Proprioception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VinT-6D, a multi-modal (vision, touch, proprioception) object-in-hand pose dataset with 2M simulated (VinT-Sim) and 0.1M real (VinT-Real) samples, plus a baseline VinT-Net fusing vision and touch.
Problem
Large-scale, real-world-aligned datasets for 6D object-in-hand pose estimation with multi-fingered hands are scarce; existing sim data has a large sim2real gap and real data is small and limited to fingertip touch or two-finger grippers.
Approach
Combines MuJoCo grasp simulation with calibrated whole-hand taxel distributions and Blender photorealistic rendering for sim data, and a motion-capture-tracked real platform with custom marker fixtures for sub-millimeter pose ground truth; a baseline VinT-Net fuses color, depth and touch point clouds for keypoint-based pose estimation.
Key results
- VinT-Real training with vision+touch improves ADD(S) AUC over vision-only on the tomato soup can (94.65 vs 86.78) and other objects.
- On tomato soup can, VinT-Net reaches 82.43 ADD-0.05d vs 76.74 for Object-Hand-Pose and 74.60 for PVN3D.
- Under 50% hand occlusion, vision+touch keeps 88.81% vs 80.43% for vision only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add touch and proprioception to vision (ablation) | VinT-Net · VinT-Real objects | ADD(S) AUC: 90.34 -> 93.45 | Helped |
| Add touch and proprioception to vision (ablation) | VinT-Net · Tomato soup can | ADD-0.05d: 86.78 -> 94.65 | Helped |
| Add touch and proprioception to vision (ablation) | VinT-Net · Tomato soup can, 20%–50% occlusion | ADD-0.05d: 93.31 -> 94.76; 80.43 -> 88.81 | Helped |
| Train on combined VinT-Sim and VinT-Real rather than either split alone | VinT-Net · VinT-6D evaluation objects | ADD(S) AUC: 87.52 -> 94.15 | Helped |
| Integrate touch data into the semantic segmentation head (ablation) | VinT-Net · Object-in-hand pose estimation | Model accuracy | No clear effect |
Surprising findings
- Integrating touch data into the semantic segmentation head did not improve performance, which the authors attribute to interference.
Limitations
- Limited diversity in objects and scenes (25 objects for sim, 10 for real).
- VinT-Net is a simple instance-level fusion baseline; category-level or zero-shot settings unexplored.
Experimental setup
Datasets: VinT-Sim VinT-Real YCB-Video
Benchmarks: VinT-6D
Metrics: ADD(S) AUC ADD-0.05d
Baselines: PVN3D Object-Hand-Pose vision-only VinT-Net
Builds on: PVN3D SAM PointNet++
Closest papers by meaning
- VTDexManip: A Dataset and Benchmark for Visual-tactile Pretraining and Dexterous Manipulation with Reinforcement Learning (ICLR 2025 · similarity 0.89)
- HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing (NeurIPS 2026 · similarity 0.88)
- EgoTac: In-the-wild Tactile Prediction from Egocentric Vision (NeurIPS 2026 · similarity 0.88)
- Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper (NeurIPS 2025 · similarity 0.87)
- ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation (NeurIPS 2025 · similarity 0.85)
- Scaffolding Dexterous Manipulation with Vision-Language Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.