Research map / Vision-language and multimodal / Human and embodied modeling
HOIGPT: Learning Long-Sequence Hand-Object Interaction with Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HOIGPT, a Flan-T5-based language model that bidirectionally maps 3D hand-object interaction sequences and text via a decomposed VQ-VAE tokenizer with geometric losses.
Problem
Generating and captioning long 3D hand-object interaction (HOI) sequences from text or partial sequences, where diffusion models struggle with flexible conditioning and long multi-action sequences.
Approach
A HOI-decomposed VQ-VAE tokenizer that factorizes hand and object motion into separate codebooks (single shared hand codebook with handedness, plus object codebook) with penetration and contact geometric losses, letting an LLM process HOI tokens alongside text.
Key results
- Text-to-HOI generation FID 3.29 vs 5.85 for Text2HOI and R Precision Top-3 74.80 vs 72.34.
- HOI prediction FID 4.08 vs 8.23 for MotionGPT; HOI interpolation FID 2.91 vs 6.16.
- Object-motion-conditioned generation improves FID from 3.29 to 2.14.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HOIGPT instead of Text2HOI | HOI generation · HOI generation | FID: 5.85 ± 0.46 -> 3.29 ± 0.54 | Helped |
| Use HOIGPT instead of MotionGPT | HOI prediction · HOI prediction | FID: 8.23 -> 4.08 | Helped |
| Use HOIGPT instead of MotionGPT | HOI interpolation · HOI interpolation | FID: 6.16 -> 2.91 | Helped |
| Replace the HOI-decomposed VQ-VAE with separate hand and object VQ-VAEs (ablation) | HOI Tokenizer · HOI tokenizer reconstruction | FID: 0.43 -> 0.87 | Hurt |
| Remove HOI-decomposed VQ (ablation) | HOI Tokenizer · HOI tokenizer reconstruction | FID: 0.43 -> 0.53 | Hurt |
| Remove penetration loss (ablation) | HOI Tokenizer · HOI tokenizer reconstruction | IV: 8.60 -> 12.32 | Hurt |
| Remove contact loss (ablation) | HOI Tokenizer · HOI tokenizer reconstruction | FID: 0.43 -> 0.67 | Hurt |
| Condition on object motion (ablation) | HOIGPT · HOI generation conditioned on object motion | FID: 3.29 -> 2.14 | Helped |
| Condition on object motion (ablation) | HOIGPT · HOI generation conditioned on object motion | RPrecision: 74.80 -> 82.71 | Helped |
Surprising findings
- The naive VQ-VAE tokenizer without decomposition yields the weakest reconstruction, while removing the decomposition still beats it in some metrics (e.g., w/o HOI-decomposed VQ FID 0.53 vs full 0.43 is worse, but diversity differs).
Limitations
- Experiments limited to ARCTIC and GRAB with few objects; scaling to more diverse datasets is future work.
- Text descriptions for GRAB were generated automatically from intent labels and object names.
Experimental setup
Datasets: ARCTIC GRAB
Benchmarks: ARCTIC GRAB
Metrics: FID Diversity R Precision MMDist MModality interpenetration volume ADE FDE
Baselines: T2MGPT MotionGPT TM2T Text2HOI
Builds on: VQ-VAE T2M-GPT MotionGPT Text2HOI MAE masking
Closest papers by meaning
- Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.90)
- FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation (NeurIPS 2026 · similarity 0.88)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.88)
- OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model (NeurIPS 2025 · similarity 0.88)
- HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data (CVPR 2024 · similarity 0.88)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.