Research map / 3D vision / Human pose estimation
Collaborative Learning for 3D Hand-Object Reconstruction and Compositional Action Recognition from Egocentric RGB Videos Using Superquadrics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-branch transformer framework using superquadrics as a template-free 3D object representation to jointly reconstruct hand-object poses and recognise verb-noun compositional actions from egocentric RGB video, evaluated on H2O and FPHA with new compositional splits.
Problem
Joint 3D hand-object pose estimation and egocentric action recognition struggle to generalise to unseen objects: bounding-box object representations are limited, object templates are needed at test time, and appearance bias hurts compositional generalisation to new verb-noun combinations.
Approach
Use superquadrics (11-parameter primitives) as an intermediate template-free 3D object representation, and a collaborative appearance/geometric two-branch transformer where verb and noun predictors use the geometry-aware verb token and appearance token, enabling compositional reasoning without branch stacking.
Key results
- On H2O official split, the method reaches 92.25% top-1 action accuracy versus 90.90% for H2OTR and 90.08% for MViTv2.
- On the harder H2O compositional split S2, the method reaches 69.93% accuracy, while MViTv2 drops by 37.76 points relative to the official split.
- On FPHA, the method achieves 98.74% top-1 accuracy, slightly above H2OTR's 98.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the collaborative learning framework with superquadrics | H2O official action recognition · H2O | top-1 accuracy: 90.90 -> 92.25 | Helped |
| add the collaborative learning framework with superquadrics | FPHA official action recognition · FPHA | top-1 accuracy: 98.4 -> 98.74 | Helped |
| add the collaborative learning framework with superquadrics | unseen-object action recognition · FPHA | top-1 accuracy (%): 76.44 -> 85.41 | Helped |
| add superquadrics (ablation) | HTT · H2O S1 | top-1 accuracy (%): 71.13 -> 73.25 | Helped |
| add superquadrics (ablation) | HTT · H2O S2 | top-1 accuracy (%): 59.88 -> 62.45 | Helped |
| add superquadrics (ablation) | Ours · H2O S1 | top-1 accuracy (%): 75.39 -> 80.59 | Helped |
| add superquadrics (ablation) | Ours · H2O S2 | top-1 accuracy (%): 65.16 -> 69.93 | Helped |
| add superquadrics (ablation) | Ours · H2O S1 | hand error (mm): 34.6 -> 31.8 | Helped |
| add superquadrics (ablation) | Ours · H2O S2 | hand error (mm): 37.7 -> 33.4 | Helped |
| remove the geometric branch (ablation) | collaborative learning framework · H2O | action accuracy | Hurt |
| remove inter-branch gradient flow (ablation) | two-branch collaborative framework · H2O | action accuracy | Hurt |
| remove either the verb or noun classifier (ablation) | collaborative learning framework · H2O | action accuracy | Hurt |
Surprising findings
- A pure appearance-based MViTv2 baseline performs comparably to state-of-the-art geometric and unified hand-object methods on the official splits.
- Removing gradient flow between the two branches yields the lowest two-branch performance, indicating unstable training without collaborative connections.
Limitations
- Relies on the expressiveness of superquadrics; non-convex objects such as cups, paper and clothes are hard to capture.
- Multi-superquadric recovery is needed for complex shapes, and superquadric quality depends on object template quality (broken meshes degrade fitting).
Experimental setup
Datasets: H2O FPHA ObMan ImageNet Kinetics-400 YCB ShapeNet
Benchmarks: H2O FPHA ObMan
Metrics: top-1 accuracy MEPE (mm) object Chamfer distance penetration depth intersection volume
Baselines: MViTv2 HTT H2OTR H+O C2D I3D SlowFast
Builds on: MANO HTT H2OTR MViTv2 EMS superquadric recovery
Closest papers by meaning
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.87)
- Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning (ICLR 2025 · similarity 0.87)
- ComPose: When to Trust Hands for Object Pose Tracking (NeurIPS 2026 · similarity 0.86)
- MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation (NeurIPS 2025 · similarity 0.86)
- SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit Meshes (NeurIPS 2025 · similarity 0.86)
- How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.