Research map / Detection and segmentation / CNN features
QORT-Former: Query-optimized Real-time Transformer for Understanding Two Hands Manipulating Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QORT-Former, a real-time Transformer that estimates 3D poses of two hands and an object from a single RGB image using 108 queries, one decoder, contact-map-enhanced queries, and a three-step image/query co-update.
Problem
Joint 3D pose estimation of two hands and a manipulated object from RGB is accurate in recent Transformer models but too slow for real-time AR/VR use, mainly due to heavy encoders, many queries, and many decoder layers.
Approach
Splits a small set of queries into left-hand, right-hand, and object groups chosen by learned query proposal modules, enriches them with estimated hand contact maps, and co-updates enhanced image features and query features in a single lightweight decoder layer using a three-step update.
Key results
- On H2O, mean end-point error improves over prior SOTA by 17.6% (left hand), 22.8% (right hand), and 27.2% (object); on FPHA by 5.3% (right hand) and 10.4% (object).
- Runs at 53.5 FPS on an RTX 3090Ti using 108 queries and 1 decoder, versus 26 FPS for H2OTR.
- Interaction recognition top-1 accuracy on H2O rises from 90.9 (H2OTR pipeline) to 91.3 when using QORT-Former as pose estimator.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hand-object queries instead of omitting them (ablation) | QORT-Former · H2O test, left hand | mean end-point error (mm): 27.3 -> 22.6 | Helped |
| Use hand-object queries instead of omitting them (ablation) | QORT-Former · H2O test, right hand | mean end-point error (mm): 33.5 -> 21.4 | Helped |
| Use hand-object queries instead of omitting them (ablation) | QORT-Former · H2O test, object | mean end-point error (mm): 36.5 -> 33.1 | Helped |
| Use hand-object queries instead of omitting them (ablation) | QORT-Former · FPHA test, right hand | mean end-point error (mm): 22.3 -> 15.8 | Helped |
| Use hand-object queries instead of omitting them (ablation) | QORT-Former · FPHA test, object | mean end-point error (mm): 24.2 -> 20.3 | Helped |
| Add contact map features to hand-object queries (ablation) | QORT-Former with hand-object queries · H2O test, left hand | mean end-point error (mm): 22.6 -> 20.1 | Helped |
| Add contact map features to hand-object queries (ablation) | QORT-Former with hand-object queries · H2O test, right hand | mean end-point error (mm): 21.4 -> 19.9 | Helped |
| Add contact map features to hand-object queries (ablation) | QORT-Former with hand-object queries · H2O test, object | mean end-point error (mm): 33.1 -> 32.9 | Helped |
| Add contact map features to hand-object queries (ablation) | QORT-Former with hand-object queries · FPHA test, right hand | mean end-point error (mm): 15.8 -> 14.2 | Helped |
| Add contact map features to hand-object queries (ablation) | QORT-Former with hand-object queries · FPHA test, object | mean end-point error (mm): 20.3 -> 18.8 | Helped |
| Add Enhanced Feature Update and location-based feature extraction (ablation) | QORT-Former decoder · H2O test, left hand | mean end-point error (mm): 26.4 -> 20.1 | Helped |
| Add location-based feature extraction (ablation) | QORT-Former decoder · H2O test, left hand | mean end-point error (mm): 22.3 -> 20.1 | Helped |
Limitations
- Relies on a single feature map and a lightweight encoder, which the authors note makes the encoder less feature-enriched than heavier models.
- Left/right hand confusion addressed only via query proposal losses; the method is evaluated only on H2O and FPHA.
Experimental setup
Datasets: H2O FPHA
Benchmarks: H2O FPHA
Metrics: mean end-point error (mm) top-1 accuracy FPS
Baselines: H2OTR Hasson et al. Tekin et al. Kwon et al. Wen et al. Aboukhadra et al. Cho et al. C2D I3D SlowFast HTT
Builds on: H2OTR deformable DETR PPM-FPN ChebGConv
Closest papers by meaning
- HOGSA: Bimanual Hand-Object Interaction Understanding with 3D Gaussian Splatting Based Data Augmentation (AAAI 2025 · similarity 0.86)
- How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions (CVPR 2025 · similarity 0.86)
- InteractVLM: 3D Interaction Reasoning from 2D Foundational Models (CVPR 2025 · similarity 0.85)
- GEARS: Local Geometry-aware Hand-object Interaction Synthesis (CVPR 2024 · similarity 0.84)
- MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation (NeurIPS 2025 · similarity 0.84)
- Learning Dense Hand Contact Estimation from Imbalanced Data (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.