Research map / Vision-language and multimodal / Human and embodied modeling

MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation

NeurIPS 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Text-driven human motion generation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MEgoHand, a VLM-plus-depth "cerebrum" with a DiT flow-matching "cerebellum" and temporal orthogonal filtering to generate egocentric hand-object motion from RGB, text, and initial MANO pose, trained on a curated 3.35M-frame RGB-D dataset.

Problem

Generating physically plausible egocentric hand-object motion from RGB, text and initial hand pose without object priors, while avoiding ambiguous textual guidance, complex contact-map pipelines and open-loop error accumulation.

Approach

A bi-level design pairs an Eagle-2 VLM with a monocular metric depth encoder (UniDepthV2 features, ResNet-50) as high-level context, and a DiT flow-matching policy that predicts chunks of future MANO parameters, smoothed at inference by Temporal Orthogonal Filtering; data is unified via an Inverse MANO Retargeting Network and a Virtual RGB-D Renderer.

Key results

• MEgoHand reduces in-domain MPJPE to 5.425 cm vs 7.726 cm for LatentAct, with MRE 0.123 vs 0.937 rad (86.9% reduction in mean MRE claimed).
• Zero-shot cross-domain MPJPE on ARCTIC is 7.358 cm vs 11.65 cm for LatentAct, and 5.775 cm vs 8.341 cm on HOLO.
• Removing the depth modality (MEgoHand-TI vs full) and depth supervision both degrade results; relative depth slightly outperforms metric depth in-domain (5.610 vs 5.425 MPJPE is actually worse, so metric is better in-domain).

Measured effects reported in the paper

Surprising findings

• Relative depth performs better than metric depth on ARCTIC and HOLO cross-domain sets, as metric depth is sensitive to camera parameter changes.
• Removing the contact map from LatentAct degrades performance, showing dependence on contact maps.

Limitations

• Only right-hand motion is considered.
• Performance is weakest on HOI4D due to diverse object instances.
• Metric depth is sensitive to camera parameter changes in cross-domain settings.

Experimental setup

Datasets: TACO FPHA HOI4D H2O HOT3D OakInk2 ARCTIC HOLO

Benchmarks: TACO HOI4D H2O HOT3D OakInk2 ARCTIC HOLO

Metrics: MPJPE MPJPE-PA MPVE MPVE-PA MWTE MRE

Baselines: LatentAct LatentAct-Diff LatentAct without contact map

Builds on: flow matching LatentAct Eagle-2 UniDepthV2 MANO

Closest papers by meaning

• LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion. (CVPR 2025 · similarity 0.90)
• Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling (AAAI 2024 · similarity 0.90)
• Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.89)
• GEARS: Local Geometry-aware Hand-object Interaction Synthesis (CVPR 2024 · similarity 0.89)
• Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learning (ICLR 2025 · similarity 0.89)
• HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos (CVPR 2025 · similarity 0.89)