Research map / Video / Video generation
🎧MOSPA: Human Motion Generation Driven by Spatial Audio
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the SAM spatial-audio motion dataset (~9 hours of binaural audio paired with SMPL-X motion) and MOSPA, a diffusion transformer that generates human body motion conditioned on spatial audio features, sound source location, and motion genre.
Problem
Generating realistic human motion that responds to spatial audio (binaural sound with directional and intensity cues) has been unexplored, and no dataset existed for it.
Approach
Combine handcrafted spatial audio features (MFCC, tempogram, chromagram, onset, RMS, active frames from both ears) with sound source location and motion genre in a transformer diffusion model that directly predicts clean motion, plus geometric, foot contact and trajectory losses, and a feature-extractor-based evaluation protocol.
Key results
- MOSPA reaches FID 7.981 and R-precision Top-1 0.937 on SAM, versus EDGE 13.993 / 0.886 and Bailando 168.396 / 0.077.
- Ablation: removing genre conditioning raises FID from 7.981 to 10.930; reducing latent dim to 256 raises FID to 9.226.
- User study with 25 participants preferred MOSPA over baselines on intent alignment, motion quality, and GT similarity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MOSPA instead of EDGE | Spatial audio-driven motion generation · SAM test | R-precision Top1: 0.886^{\pm 0.005} -> 0.937^{\pm 0.005} | Helped |
| Use MOSPA instead of EDGE | Spatial audio-driven motion generation · SAM test | FID: 13.993 -> 7.981 | Helped |
| Use MOSPA instead of EDGE | Spatial audio-driven motion generation · SAM test | Diversity: 23.099^{\pm 0.196} -> 23.575^{\pm 0.188} | Helped |
| Use MOSPA instead of EDGE | Spatial audio-driven motion generation · SAM test | APD: 43.882 -> 53.915 | Mixed |
| Increase latent dimension from 256 to 512 (ablation) | MOSPA · SAM test | R-precision Top1: 0.891^{\pm 0.005} -> 0.937^{\pm 0.005} | Helped |
| Increase attention heads from 4 to 8 (ablation) | MOSPA · SAM test | R-precision Top1: 0.923^{\pm 0.004} -> 0.937^{\pm 0.005} | Helped |
| Increase diffusion steps from 100 to 1000 (ablation) | MOSPA · SAM test | FID: 8.456 -> 7.981 | Helped |
| Increase diffusion steps from 4 to 1000 (ablation) | MOSPA · SAM test | FID: 8.387 -> 7.981 | Helped |
| Add motion genre conditioning (ablation) | MOSPA · SAM test | R-precision Top1: 0.889^{\pm 0.005} -> 0.937^{\pm 0.005} | Helped |
| Add MFCC features (ablation) | MOSPA · SAM test | FID: 9.070 -> 7.981 | Helped |
| Add tempogram features (ablation) | MOSPA · SAM test | FID: 10.786 -> 7.981 | Helped |
Surprising findings
- Reducing diffusion steps from 1000 to 4 barely hurts and slightly improves R-precision Top-3 (0.998 vs 0.996), with FID 8.387 vs 7.981.
Limitations
- Generated motions lack physical constraints and may be physically implausible.
- Only body motion is modeled; fingers and facial expressions are excluded.
- No awareness of surrounding scene geometry.
Experimental setup
Datasets: SAM (own dataset) AudioSet
Benchmarks: SAM
Metrics: R-precision FID Diversity APD user study preference
Baselines: EDGE POPDG LODGE Bailando
Builds on: EDGE diffusion models (x0 prediction) residual feature fusion SMPL-X
Closest papers by meaning
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.89)
- Diffusion-based Realistic Listening Head Generation via Hybrid Motion Modeling (CVPR 2025 · similarity 0.88)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.88)
- MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls (AAAI 2025 · similarity 0.88)
- AMD: Autoregressive Motion Diffusion (AAAI 2024 · similarity 0.88)
- AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformers (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.