Research map / Video / Video generation
SemGeoMo: Dynamic Contextual Human Motion Generation with Semantic and Geometric Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SemGeoMo, a two-stage diffusion framework that generates responsive human motions from sequential point clouds of dynamic objects, guided by LLM-generated text and hierarchical affordance/joint geometric cues.
Problem
Generating physically plausible, semantically coherent human interaction motions in dynamic environments where interacted objects or people move, given only their sequential point clouds.
Approach
Decouples contact geometry generation (diffusion predicting affordance maps and hand-joint positions from point clouds and text) from full-body motion generation (a ControlNet on frozen MDM with joint and foot loss guidance), and uses a LoRA-tuned LLaMA annotator to produce coarse and fine-grained text descriptions as semantic guidance.
Key results
- On FullBodyManipulation with generated text, SemGeoMo reaches HandJPE 30.35 and FID 1.05 versus OMOMO's 33.18 and 1.98 and CHOIS's 31.68 and 2.27.
- On BEHAVE, SemGeoMo achieves HandJPE 27.91 and F1 0.74, versus the best baseline OMOMO's F1 0.62.
- On the unseen HoDome dataset, SemGeoMo reaches F1 0.54 and HandJPE 44.22, versus CHOIS's F1 0.18 and HandJPE 76.74.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use generated text instead of ground-truth text | SemGeoMo · FullBodyManipulation | HandJPE: 27.84 -> 30.35 | Mixed |
| use generated text instead of ground-truth text | SemGeoMo · FullBodyManipulation | FID: 1.17 -> 1.05 | Helped |
| remove affordance-map guidance (ablation) | SemGeoMo · FullBodyManipulation | FID: 1.03 -> 2.21 | Hurt |
| remove joint guidance (ablation) | SemGeoMo · FullBodyManipulation | FID: 1.03 -> 3.52 | Hurt |
| remove cross-attention (ablation) | SemGeoMo · FullBodyManipulation | FID: 1.03 -> 9.27 | Hurt |
| remove text guidance (ablation) | SemGeoMo · FullBodyManipulation | R-score: 0.68 -> 0.41 | Hurt |
| remove fine-grained text (ablation) | SemGeoMo · FullBodyManipulation | FID: 1.03 -> 1.17 | Hurt |
| remove joint guidance loss (ablation) | SemGeoMo · FullBodyManipulation | Cprec: 0.84 -> 0.83 | Hurt |
| remove foot guidance loss (ablation) | SemGeoMo · FullBodyManipulation | FS: 0.57 -> 0.61 | Hurt |
Surprising findings
- Removing the fine-grained text annotation slightly improves some metrics (FID 1.17 vs 1.03 for full model is not improved, but R-score and FS differ), and the ground-truth-text variant performs comparably to generated text.
Limitations
- Evaluated mainly on human-object interaction datasets with limited object variety
- Relies on an LLM annotator whose text quality is validated only via BLEU/ROUGE against provided annotations
- Text annotations for BEHAVE, IMHD2, HoDome were produced by the authors' own pipeline
Experimental setup
Datasets: FullBodyManipulation BEHAVE IMHD2 HoDome
Benchmarks: FullBodyManipulation BEHAVE IMHD2 HoDome
Metrics: HandJPE MPJPE contact precision contact recall contact accuracy F1 contact percentage FID R-score diversity
Baselines: SceneDiff OMOMO MDM-PC CHOIS AffordMotion
Builds on: MDM OMOMO CHOIS AffordMotion ControlNet
Closest papers by meaning
- TIMotion: Temporal and Interactive Framework for Efficient Human-Human Motion Generation (CVPR 2025 · similarity 0.89)
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.88)
- HumanTOMATO: Text-aligned Whole-body Motion Generation (ICML 2024 · similarity 0.88)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.87)
- Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes (ICLR 2025 · similarity 0.86)
- InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.