Research map / 3D vision / 3D generation
MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MIMO synthesizes controllable character videos from a single reference image by decomposing training videos into human, scene and occlusion layers via monocular depth and encoding them as identity, structured motion (SMPL-anchored) and scene codes for a diffusion decoder.
Problem
Controllable character video synthesis in real-world scenes that needs arbitrary characters, generalization to novel 3D motions, and human-object interaction with occlusions, without per-case 3D training or 2D video annotations.
Approach
Lift 2D frames to 3D with monocular depth, split each video into main human, underlying scene and floating occlusion layers by depth, and encode them as a canonical identity code, an SMPL-anchored structured motion code, and a full scene code that conditions a Stable Diffusion-based video decoder.
Key results
- Achieves PSNR 25.210, SSIM 0.883, LPIPS 0.125, FVD 221.4 versus best baseline Champ (PSNR 21.044) and Animate Anyone (FVD 304.3).
- Ablation: removing spatial decomposed modeling drops PSNR to 22.148 and raises FVD to 268.5.
- Structured motion code outperforms 2D skeleton (PSNR 24.326) and 3D maps (24.402) alternatives.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MIMO instead of Animate Anyone | Character video synthesis · Test set | PSNR: 21.003 -> 25.210 | Helped |
| use MIMO instead of Mimic-Motion | Character video synthesis · Test set | PSNR: 20.688 -> 25.210 | Helped |
| use MIMO instead of Champ | Character video synthesis · Test set | PSNR: 21.044 -> 25.210 | Helped |
| add spatial decomposed modeling (ablation) | MIMO · Test set | PSNR: 22.148 -> 25.210 | Helped |
| use structured motion code instead of 2D skeleton (ablation) | MIMO · Test set | PSNR: 24.326 -> 25.210 | Helped |
| use structured motion code instead of 3D maps (ablation) | MIMO · Test set | PSNR: 24.402 -> 25.210 | Helped |
| add canonical appearance transfer (ablation) | MIMO · Test set | PSNR: 24.918 -> 25.210 | Helped |
| add independent occlusion encoding (ablation) | MIMO · Interactive scenes | occlusion handling | Helped |
Limitations
- Relies on multiple pretrained off-the-shelf components (depth estimator, human detector, tracker, inpainting, SMPL estimator) whose errors can propagate.
- Evaluation uses a self-collected 100-video in-the-wild test set, with no public benchmark comparison.
Experimental setup
Datasets: HUD-7K AMASS Mixamo
Benchmarks: in-the-wild human video test set (100 videos, 150-frame clips)
Metrics: PSNR SSIM LPIPS FVD
Baselines: Animate Anyone Mimic-Motion Champ MagicAnimate
Builds on: Stable Diffusion 1.5 AnimateDiff Animate Anyone SMPL En3D
Closest papers by meaning
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.88)
- Towards Multiple Character Image Animation Through Enhancing Implicit Decoupling (ICLR 2025 · similarity 0.87)
- AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion Models (CVPR 2025 · similarity 0.86)
- Animate-X: Universal Character Image Animation with Enhanced Motion Representation (ICLR 2025 · similarity 0.86)
- Follow-Your-Click: Open-domain Regional Image Animation via Motion Prompts (AAAI 2025 · similarity 0.86)
- Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.