Research map / 3D vision / 3D generation
How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Predicts 3D hand motion and contact maps (interaction trajectories) from a single RGB image, action text, and a 3D contact point, using a VQVAE codebook of interactions, a learned indexer, and a transformer interaction predictor trained on a data engine over HoloAssist.
Problem
Predicting future 3D hand poses and contact maps for everyday hand-object interactions without 3D object models, given only a single image, action text, and a 3D contact point; hands are often not visible at the start of interactions.
Approach
Learn a discrete latent codebook of hand-pose and contact trajectories with a VQVAE, then index it at test time from image, text and contact-point inputs and decode the trajectory with a transformer predictor, so that prototypical hand motions are reused across objects and tasks.
Key results
- LatentAct reduces task-level forecasting MPJPE to 7.61 cm vs 8.32 cm for HCTFormer with hand visible, and raises F1 from 0.72 to 0.75.
- In interpolation with hand visible, LatentAct reaches MPJPE 6.72 cm and F1 0.80, versus 7.52 cm and 0.76 for HCTFormer.
- On ARCTIC, LatentAct trained with the HoloAssist codebook and indexer reaches MPJPE 15.36 cm and F1 0.45, versus 15.76 cm and 0.36 for HCTFormer trained on ARCTIC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LatentAct instead of HCTFormer | Task-level forecasting, hand visible · HoloAssist | MPJPE: 8.32 -> 7.61 | Helped |
| use LatentAct instead of HCTFormer | Task-level forecasting, hand visible · HoloAssist | F1: 0.72 -> 0.75 | Helped |
| use LatentAct instead of HCTFormer | Task-level interpolation, hand visible · HoloAssist | MPJPE: 7.52 -> 6.72 | Helped |
| use LatentAct instead of HCTDiff | Task-level interpolation, hand visible · HoloAssist | MPJPE-PA: 2.72 -> 2.87 | Hurt |
| use LatentAct instead of HCTFormer | Task-level interpolation, hand visible · HoloAssist | F1: 0.76 -> 0.80 | Helped |
| use LatentAct instead of HCTFormer | Task-level forecasting, no hands · HoloAssist | MPJPE: 8.34 -> 7.93 | Helped |
| use LatentAct instead of HCTFormer | Task-level forecasting, no hands · HoloAssist | F1: 0.72 -> 0.76 | Helped |
| use LatentAct instead of HCTFormer | Task-level interpolation, no hands · HoloAssist | MPJPE: 7.30 -> 7.13 | Helped |
| use LatentAct instead of HCTFormer | Task-level interpolation, no hands · HoloAssist | F1: 0.76 -> 0.79 | Helped |
| train with ARCTIC LatentAct rather than HCTFormer | ARCTIC-trained models · ARCTIC | MPJPE: 15.76 -> 14.77 | Helped |
| add InterCode (ablation) | HCTFormer · Task-level interpolation, hand visible | MPJPE: 7.52 -> 6.72 | Helped |
| add InterCode (ablation) | HCTDiff · Task-level interpolation, hand visible | MPJPE: 8.30 -> 6.53 | Helped |
Surprising findings
- Diffusion-based interaction predictor (LatentAct-Diff) improves hand pose error but worsens contact map F1 relative to the transformer variant.
- Zero-shot transfer from HoloAssist to ARCTIC is competitive with models trained directly on ARCTIC in MPJPE, though its F1 is low (0.19).
Limitations
- Does not predict object state changes and assumes 3D object models are unavailable.
- Annotations are programmatically generated from HaMeR and SAMv2 and may be inaccurate in some cases.
- Evaluation measures accuracy against a single ground truth and does not account for multimodal futures.
Experimental setup
Datasets: HoloAssist ARCTIC
Benchmarks: HoloAssist-derived interaction trajectory benchmark ARCTIC
Metrics: MPJPE MPJPE-PA F1
Baselines: HCTFormer HCTDiff ARCTIC-trained HCTFormer
Builds on: VQVAE MDM MANO HaMeR SAMv2
Closest papers by meaning
- Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.92)
- Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling (AAAI 2024 · similarity 0.90)
- LatentHOI: On the Generalizable Hand Object Motion Generation with Latent Hand Diffusion. (CVPR 2025 · similarity 0.90)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.89)
- GEARS: Local Geometry-aware Hand-object Interaction Synthesis (CVPR 2024 · similarity 0.89)
- HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.