Research map / 3D vision / 3D generation
AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AvatarGO, a zero-shot framework that composites a generated 3D human and object using LLM/Lang-SAM-guided contact retargeting and then animates them with correspondence-aware motion optimization via SMPL-X skinning to generate 4D human-object interactions from text.
Problem
Generating animatable 4D human-object interaction scenes from text is hard because diffusion models poorly capture where objects contact the human body and how to animate both without penetration; existing methods are SMPL-based or treat the scene as one entity.
Approach
Uses Lang-SAM to localize the contact body part from the prompt and initialize the object's translation on the human's Gaussians, then optimizes object rotation/translation with a correspondence loss that ties human and object SMPL-X skinning fields together, so the object follows the body during animation.
Key results
- Outperforms DreamGaussian4D, HumanGaussian, TC4D and variants in qualitative comparisons and user studies (24 volunteers, 1-5 scale) on all seven criteria.
- Reports CLIP-based metrics where AvatarGO maintains human identity in composited scenes, while Var-A and GraphDreamer score slightly higher on CLIP-Image.
- Training takes about 10 min for the 3D stage and 20 min for the 4D stage on a single NVIDIA A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Lang-SAM contact-area initialization (ablation) | AvatarGO 3D composition · 3D human-object composition | contact accuracy | Helped |
| Use correspondence-aware loss instead of SDF distance loss (ablation) | AvatarGO motion optimization · 4D animation | spatial relationship preservation | Helped |
| Use correspondence-aware loss instead of SDF label loss (ablation) | AvatarGO motion optimization · 4D animation | spatial relationship preservation | Helped |
| Remove trainable rotation, translation, and correspondence-aware loss (ablation) | AvatarGO motion optimization · 4D animation | penetration robustness | Hurt |
| Remove correspondence-aware loss and train with SDS losses only (ablation) | AvatarGO motion optimization · 4D animation | penetration robustness | Hurt |
| Remove spatial-aware SDS (ablation) | AvatarGO · 3D composition | optimization of object rotation, scale, and translation | Hurt |
| Remove spatial-aware SDS (ablation) | AvatarGO · 4D animation | animated avatar quality | Hurt |
| Replace correspondence-aware motion supervision with trajectory-guided object motion and video-diffusion SDS (ablation) | AvatarGO (Var-B) · 4D animation | human-object relationship preservation | Hurt |
| Use direct SMPL LBS animation | HumanGaussian · 4D animation | motion smoothness | Hurt |
| Animate the entire composited scene with DreamGaussian4D | DreamGaussian4D · 4D animation | interaction animation quality | Hurt |
| Treat the scene as a single entity during animation | TC4D · 4D animation | individual-object motion | Hurt |
| Use graph-based relationships without explicit contact retargeting | GraphDreamer · 3D human-object composition | contact accuracy | Hurt |
Surprising findings
- Var-A and GraphDreamer score slightly higher on CLIP-Image than AvatarGO, attributed to them failing to perform the composition.
Limitations
- Assumes rigid-body objects, so non-rigid content such as flags is unsupported.
- Assumes continuous human-object contact, so intermittent interactions like dribbling a basketball are not handled.
- Penetration can still occur when the object is close to the body under the contact-part-only baseline.
Experimental setup
Benchmarks: AvatarGO human-object pairs (self-collected)
Metrics: CLIP-Score CLIP-Image similarity CLIP directional similarity user study ratings (penetration, scale, contact, motion qual
Baselines: DreamGaussian4D HumanGaussian TC4D GraphDreamer Comp4D-style variant (Var-B) Var-A
Builds on: DreamGaussian4D DreamGaussian ComboVerse spatial-aware SDS HumanGaussian SMPL-X
Closest papers by meaning
- CG-HOI: Contact-Guided 3D Human-Object Interaction Generation (CVPR 2024 · similarity 0.89)
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.89)
- InteractAnything: Zero-shot Human Object Interaction Synthesis via LLM Feedback and Object Affordance Parsing (CVPR 2025 · similarity 0.89)
- Morphable Diffusion: 3D-Consistent Diffusion for Single-image Avatar Creation (CVPR 2024 · similarity 0.89)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.89)
- InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.