Research map / Video / Video generation
InterMask: 3D Human Interaction Generation via Collaborative Masked Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InterMask, a collaborative masked generative transformer over 2D discrete motion token maps from a VQ-VAE, for text-conditioned two-person interaction generation, and extends it to reaction generation without fine-tuning.
Problem
Generating realistic and text-faithful 3D two-person human interactions from text; existing diffusion-based methods (InterGen, in2IN, ComMDM) lack realism and fidelity.
Approach
Replace 1D motion tokens with a 2D spatio-temporal token map from a per-individual VQ-VAE, and model both individuals' tokens jointly with a masked transformer that uses spatio-temporal self/cross attention, two-stage (random or interaction) masking, and step-unroll masking.
Key results
- Achieves FID 5.154 on InterHuman (vs 5.535 for in2IN) and 0.399 on InterX (vs 5.207 for InterGen).
- User study with 16 participants: interaction quality 4.089 vs 3.296 for InterGen; 69.14% preferred InterMask.
- Reaction generation on InterHuman reaches FID 2.99 and R-precision 0.462 without fine-tuning, while InterGen degrades to FID 52.89.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InterMask instead of in2IN | Text-conditioned interaction generation · InterHuman test | FID: 5.535 -> 5.154 | Helped |
| Use InterMask instead of InterGen | Text-conditioned interaction generation · InterX test | FID: 5.207 -> 0.399 | Helped |
| Use InterMask instead of InterGen | Text-conditioned interaction generation · InterX test | R-Precision Top 1: 0.207 -> 0.403 | Helped |
| Use InterMask instead of InterGen | Reaction generation without fine-tuning · InterHuman reaction generation | FID: 52.89 -> 2.99 | Helped |
| Use InterMask instead of InterGen | Reaction generation without fine-tuning · InterHuman reaction generation | R-Precision: 0.194 -> 0.462 | Helped |
| Generate tokens one person at a time rather than collaboratively (ablation) | Inter-M Transformer · InterHuman test | FID: 5.154 -> 7.637 | Hurt |
| Remove self-attention (ablation) | Inter-M Transformer · InterHuman test | FID: 5.154 -> 5.715 | Hurt |
| Remove cross-attention (ablation) | Inter-M Transformer · InterHuman test | FID: 5.154 -> 8.461 | Hurt |
| Remove spatio-temporal attention (ablation) | Inter-M Transformer · InterHuman test | FID: 5.154 -> 10.968 | Hurt |
| Remove step unroll and interaction masking (ablation) | Inter-M Transformer · InterHuman test | FID: 5.154 -> 5.629 | Hurt |
| Set random masking probability p_r to 0.7 rather than 0.8 (ablation) | InterMask masking strategy · InterHuman interaction generation | FID: 5.154 -> 5.214 | Hurt |
| Set random masking probability p_r to 0.9 rather than 0.8 (ablation) | InterMask masking strategy · InterHuman interaction generation | FID: 5.154 -> 5.152 | No clear effect |
Surprising findings
- Diversity-oriented alternative (one-person-at-a-time) modeling yields higher MModality (2.192) than InterMask but much worse FID (7.637 vs 5.154).
- InterGen applied zero-shot to reaction generation degrades sharply (FID 52.89), indicating it cannot do reaction generation out of the box.
Limitations
- Body penetration between individuals when converted to SMPL meshes.
- Implicit bias toward dancing without explicit prompting.
- Optimized for short sequences up to 10 seconds.
Experimental setup
Datasets: InterHuman InterX AMASS SMPL-X
Benchmarks: InterHuman InterX
Metrics: FID R-precision MM Dist Diversity MModality user study preference
Baselines: T2M MDM ComMDM InterGen MoMat-MoGen in2IN
Builds on: MoMask MMM T2M-GPT InterGen DiT-style AdaLN modulation
Closest papers by meaning
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.90)
- Learning to Generate Human-Human-Object Interactions from Textual Descriptions (NeurIPS 2025 · similarity 0.88)
- InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint (NeurIPS 2024 · similarity 0.87)
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.87)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.87)
- MMM: Generative Masked Motion Model (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.