Research map / Video / Video generation
SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SViMo, a synchronized diffusion model that jointly generates hand-object interaction videos and 3D hand/object motion by extending CogVideoX-5B, with a vision-aware 3D interaction diffusion module forming a closed-loop feedback.
Problem
3D HOI motion generation depends on predefined object models and lab-captured data, limiting generalization, while HOI video generation sacrifices physical plausibility and motion consistency.
Approach
Jointly denoise video latents and rendered-motion video latents in one DiT with triple-modality adaptive modulation and 3D full attention, and add a vision-aware 3D interaction diffusion model whose predicted 3D hand/object trajectories are fed back as interaction guidance and gradient constraints.
Key results
- Overall VBench score of 0.8785 on TACO, best among compared video models (CogVideoX-5B 0.8727, Wan-14B 0.7675).
- Motion MPJPE 0.1087 vs 0.3255 for EMDM and Chamfer distance 0.1577 vs 0.7788 for EMDM on TACO motion generation.
- User study: 78.42% preference rate for generated videos over baselines (1,066 responses from 41 participants).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add gradient constraints to interaction guidance (ablation) | SViMo with VID · TACO test | Overall: 0.8770 -> 0.8800 | Helped |
| add interaction guidance to gradient constraints (ablation) | SViMo with VID · TACO test | Overall: 0.8723 -> 0.8800 | Helped |
| add VID interaction guidance and gradient constraints (ablation) | SViMo · TACO test | FID: 0.0546 -> 0.0100 | Helped |
| generate video and motion independently, conditioning video on predicted motion (ablation) | SViMo · TACO test | Overall: 0.8719 -> 0.8381 | Hurt |
| generate video and motion independently, conditioning video on predicted motion (ablation) | SViMo · TACO test | FID: 0.0546 -> 0.0575 | Hurt |
| apply motion modulation only to even-numbered DiT blocks (ablation) | SViMo · TACO test | Overall: 0.8800 -> 0.8783 | Hurt |
| apply motion modulation only to even-numbered DiT blocks (ablation) | SViMo · TACO test | FID: 0.0100 -> 0.1368 | Hurt |
| use LoRA training instead of full-parameter fine-tuning (ablation) | SViMo · TACO test | Overall: 0.8800 -> 0.8631 | Hurt |
| use LoRA training instead of full-parameter fine-tuning (ablation) | SViMo · TACO test | FID: 0.0100 -> 0.4671 | Hurt |
| use SViMo rather than CogVideoX-5B for HOI video generation | TACO test video generation · TACO test | Overall: 0.8727 -> 0.8785 | Helped |
Surprising findings
- Generating video and motion independently (VModel w/ Pred. Mot.) lowers the video overall score from 0.8719 to 0.8381, showing joint modeling matters.
Limitations
- Object point clouds restricted to rigid, simple objects and struggle with complex geometry.
- Relies on a large video foundation model; LoRA fine-tuning gives suboptimal results.
- Failure cases include penetration, low dynamics, and object inconsistency.
Experimental setup
Datasets: TACO
Benchmarks: TACO VBench metrics
Metrics: Subject Consistency Background Consistency Temporal Smoothness Dynamic Degree Overall MPJPE Motion Smoothness Chamfer Distance FID
Baselines: Hunyuan-13B Wan-14B Animate Anyone EasyAnimate CogVideoX-5B MDM EMDM
Builds on: CogVideoX MDM T5 VBench DeepSpeed ZeRO
Closest papers by meaning
- InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation (CVPR 2025 · similarity 0.92)
- Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Model (CVPR 2025 · similarity 0.90)
- HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awareness (NeurIPS 2024 · similarity 0.89)
- HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models (AAAI 2025 · similarity 0.89)
- HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data (CVPR 2024 · similarity 0.89)
- Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.