Research map / 3D vision / 3D generation
TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TACO, a 2.5K-sequence real-world bimanual tool-action-object hand-object interaction dataset with an automatic mocap-plus-multiview annotation pipeline, and benchmarks compositional action recognition, hand-object motion forecasting, and cooperative grasp synthesis.
Problem
Existing hand-object interaction datasets focus on unimanual actions or single objects, limiting study of generalizable bimanual tool-action-object understanding to unseen geometries and novel triplets.
Approach
Combine marker-based optical mocap with multi-view markerless sensing in a fully automatic pipeline (object pose from markers, MANO hand fitting with contact-aware losses, SAM/Track-Anything segmentation, LaMa marker inpainting) to build a large bimanual dataset organized by tool-action-object triplets with controlled generalization splits.
Key results
- Compositional action recognition: CACNF reaches 63.02% Top-1 on S3 vs AIM 53.65%, and 44.00% vs 39.33% on S4 (compound generalization), versus about 83-86% on non-generalizable S1.
- Cross-dataset hand pose: combining TACO and DexYCB training improves CMR and MobRecon on both datasets (e.g., MobRecon DexYCB MPJPE 15.17 to 12.95 mm when adding TACO... reported as 8.69/20.14 PA-MPJPE on TACO with combined training).
- Marker removal: U-Net marker segmentation mIoU drops from 63.8% on raw images to 11.1% on marker-removed images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CACNF instead of AIM for compositional action recognition | Compositional action recognition · S3 and S4 | Top-1 accuracy: 63.02 vs 53.65; 44.00 vs 39.33 | Helped |
| Use bounding-box-based compositional recognition (CACNF) instead of AIM | Compositional action recognition · S3 and S4 | Top-5 accuracy: 92.97 vs 81.25; 80.50 vs 73.67 | Helped |
| Test on novel tool geometries within known tool categories | Compositional action recognition · S2 | Top-1 accuracy: AIM: 83.08 -> 82.81 | No clear effect |
| Test on novel tool-action-object triplets | Compositional action recognition · S3 | Top-1 accuracy: AIM: 83.08 -> 53.65; CACNF: 86.15 -> 63.02 | Hurt |
| Add interaction-environment inputs to the grasp synthesizer (ablation) | HALO-VAE · S1 & S3 | Contact ratio (%): 76.27 -> 82.86 | Helped |
| Add interaction-environment inputs to the grasp synthesizer (ablation) | HALO-VAE · S1 & S3 | Collision ratio (%): 4.19 -> 2.39 | Helped |
| Add interaction-environment inputs to the grasp synthesizer (ablation) | HALO-VAE · S2 & S4 | Contact ratio (%): 71.69 -> 83.52 | Helped |
| Add interaction-environment inputs to the grasp synthesizer (ablation) | HALO-VAE · S2 & S4 | Collision ratio (%): 14.25 -> 11.11 | Helped |
| Generalize to unseen tool geometries or categories | Cooperative grasp synthesis · S2 & S4 vs S1 & S3 | Collision ratio (%): HALO-VAE: 2.39 -> 11.11 | Hurt |
| Generalize to unseen tool geometries or categories | Cooperative grasp synthesis · S2 & S4 vs S1 & S3 | Contact ratio (%): HALO-VAE: 82.86 -> 83.52; HALO-VAE−: 76.27 -> 71.69 | Mixed |
Surprising findings
- Generative forecasting models (InterVAE, MDM) produce much larger errors than predictive models (InterRNN, CAHMP) for hands and tool.
- Compound generalization with novel tool categories is hardest for action recognition, with accuracy falling to about 40-44% Top-1.
Limitations
- Does not cover articulated objects.
- Lacks scene diversity.
- Marker removal via generative inpainting means original object appearance cannot be recovered perfectly.
Experimental setup
Datasets: TACO DexYCB
Benchmarks: TACO
Metrics: Top-1 accuracy Top-5 accuracy MPJPE PA-MPJPE Je Te Re interpenetration volume contact ratio collision ratio
Baselines: AIM CACNF InterVAE MDM InterRNN CAHMP ContactGen HALO-VAE
Builds on: MANO Segment Anything Model Track-anything LaMa CVAE
Closest papers by meaning
- OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion (CVPR 2024 · similarity 0.89)
- GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities (CVPR 2025 · similarity 0.88)
- ManiVideo: Generating Hand-Object Manipulation Video with Dexterous and Generalizable Grasping (CVPR 2025 · similarity 0.87)
- BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects (CVPR 2025 · similarity 0.87)
- LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion. (CVPR 2025 · similarity 0.87)
- GEARS: Local Geometry-aware Hand-object Interaction Synthesis (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.