Research map / Detection and segmentation / Tracking
HOIST-Former: Hand-held Objects Identification Segmentation and Tracking in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOIST-Former, a Mask2Former-based transformer that jointly segments and tracks hand-held objects by mutually pooling hand and object queries, plus a contact loss and a new in-the-wild HOIST video dataset.
Problem
Identifying, segmenting, and tracking hand-held objects in unconstrained videos, where objects are intermittently held, released, and re-picked under occlusion and rapid motion, which open-vocabulary category-based VIS methods do not handle.
Approach
A Hand-Object Transformer decoder alternates mask-attention and cross-attention between hand queries and object queries so object masks are conditioned on hand appearance and hand masks on held objects, with a contact loss focusing on hand-object contact regions; the output mask is empty when the object is not held.
Key results
- HOIST-Former achieves 56.4 AP on the HOIST test set, versus 51.8 for Mask2Former and 48.9 for the best oracle-based baseline BL-A.
- On VISOR, HOIST-Former reaches 46.6 AP (mask) vs 42.8 for Mask2Former; on UVO it reaches 66.7 vs 63.1.
- Ablations show removing hand-to-object attention drops AP to 52.7, object-to-hand attention to 53.5, and contact loss to 54.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HOIST-Former instead of Mask2Former | hand-held object segmentation and tracking · HOIST test | AP: 51.8 -> 56.4 | Helped |
| Use HOIST-Former instead of Mask2Former | hand-held object segmentation and tracking · VISOR | AP: 42.8 -> 46.6 | Helped |
| Use HOIST-Former instead of Mask2Former | hand-held object segmentation and tracking · UVO | AP: 63.1 -> 66.7 | Helped |
| add Hand-to-Object Attention (ablation) | HOIST-Former · HOIST test | AP: 52.7 -> 56.4 | Helped |
| add Object-to-Hand Attention (ablation) | HOIST-Former · HOIST test | AP: 53.5 -> 56.4 | Helped |
| add Contact Loss (ablation) | HOIST-Former · HOIST test | AP: 54.8 -> 56.4 | Helped |
Surprising findings
- Automatic pipelines using the 100DOH detector for initial boxes halve AP relative to ground-truth boxes, showing hand-held object detection is the bottleneck.
Limitations
- Fails to segment thin or partially occluded object parts.
- Can assign a previously used ID to a different object after reappearance.
- UVO evaluation uses only 80 manually selected videos, too few for training.
Experimental setup
Datasets: HOIST VISOR UVO 100DOH
Benchmarks: HOIST VISOR UVO
Metrics: Average Precision (AP) IoU
Baselines: Mask2Former STCN SAM with ByteTrack SAM with StrongSORT SAM with MixFormer SAM with GTR 100DOH detector IoUTracker
Builds on: Mask2Former Segment Anything STCN
Closest papers by meaning
- HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction (NeurIPS 2025 · similarity 0.88)
- HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness (NeurIPS 2024 · similarity 0.85)
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild (CVPR 2025 · similarity 0.84)
- ComPose: When to Trust Hands for Object Pose Tracking (NeurIPS 2026 · similarity 0.84)
- HOI-M^3: Capture Multiple Humans and Objects Interaction within Contextual Environment (CVPR 2024 · similarity 0.84)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.