Research map / Video / Video generation
Auto-Regressive Diffusion for Generating 3D Human-Object Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARDHOI, an autoregressive diffusion model over a contrastive VAE continuous token space with a Mamba context encoder and MLP denoiser for text-driven human-object interaction generation.
Problem
Text-to-HOI generation struggles to keep interactions consistent over long sequences, and discrete VQ tokenization generalizes poorly on small HOI datasets with physically implausible contacts.
Approach
A contrastive VAE (triplet loss with contact-distance-based positive/negative samples) learns a physically plausible continuous HOI token space; an autoregressive diffusion model with a Mamba context encoder and lightweight MLP denoiser predicts the next token, avoiding quantization.
Key results
- On OMOMO, ARDHOI achieves FID 0.826 vs 1.075 for HOI-Diff (best baseline), R-precision top-1 0.628.
- On BEHAVE, ARDHOI achieves FID 1.872 vs 1.998 for HOI-Diff.
- Inference takes 1.25 sec per sentence vs 75.3 sec for HOI-Diff and 29.2 sec for MDM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ARDHOI instead of HOI-Diff | Text-to-HOI generation · OMOMO | FID: 1.075 -> 0.826 | Helped |
| use ARDHOI instead of HOI-Diff | Text-to-HOI generation · BEHAVE | FID: 1.998 -> 1.872 | Helped |
| use token size 16 rather than token size 1 (ablation) | ARDHOI · OMOMO | FID: 1.289 -> 0.826 | Helped |
| use token size 16 rather than token size 2 (ablation) | ARDHOI · OMOMO | FID: 1.267 -> 0.826 | Helped |
| use token size 16 rather than token size 4 (ablation) | ARDHOI · OMOMO | FID: 1.306 -> 0.826 | Helped |
| use token size 16 rather than token size 8 (ablation) | ARDHOI · OMOMO | FID: 1.023 -> 0.826 | Helped |
| use token size 16 rather than token size 10 (ablation) | ARDHOI · OMOMO | FID: 0.929 -> 0.826 | Helped |
| use token size 16 rather than token size 12 (ablation) | ARDHOI · OMOMO | FID: 0.886 -> 0.826 | Helped |
| use token size 16 rather than token size 20 (ablation) | ARDHOI · OMOMO | FID: 0.977 -> 0.826 | Helped |
| use token size 16 rather than token size 24 (ablation) | ARDHOI · OMOMO | FID: 1.084 -> 0.826 | Helped |
| add triplet loss to the cVAE (ablation) | ARDHOI · OMOMO | FID: 0.948 -> 0.826 | Helped |
| use Mamba instead of Transformer encoder blocks for context (ablation) | ARDHOI · OMOMO | FID: 0.979 -> 0.826 | Helped |
Surprising findings
- Transformer-based VAE is not ideal for mapping entire HOI sequences into latent space, per the authors.
- On OMOMO, the R-precision top-1 of ARDHOI (0.628) is below the ground truth top-1 (0.663) but the paper also notes InterDiff gets higher top-1 in the initial-state setting than ARDHOI? (not applicable)
Limitations
- Other recent HOI methods could not be reproduced or obtained, so comparisons are limited to MDM, InterDiff and HOI-Diff.
- Positive and negative sample construction relies on hand-chosen thresholds (tau=3cm) and margin alpha that must be carefully tuned.
Experimental setup
Datasets: OMOMO BEHAVE
Benchmarks: OMOMO BEHAVE
Metrics: FID R-precision MMD Diversity AITS FLOPs
Baselines: MDM InterDiff HOI-Diff Retrieval
Builds on: Mamba MDM HOI-Diff DDIM classifier-free guidance PointNet++
Closest papers by meaning
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.92)
- HOIAnimator: Generating Text-prompt Human-object Animations using Novel Perceptive Diffusion Models (CVPR 2024 · similarity 0.90)
- Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression (CVPR 2025 · similarity 0.89)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.89)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.89)
- HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusion (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.