Research map / Video / Video generation
MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MotionStone, an image-to-video diffusion transformer conditioned on separately estimated object and camera motion intensities, where the estimator is trained from human pairwise motion comparisons plus tracking-based pseudo-labels.
Problem
Image-to-video generation lacks a reliable, human-aligned estimator of motion intensity that separates object motion from camera motion, so motion-aware models cannot be trained or controlled well on in-the-wild video.
Approach
A TAdaConv-based motion estimator with two MLP heads (object and camera) trained with pairwise ranking loss on 5,000 human-labeled video pairs plus regression on tracking pseudo-labels; the predicted scores are injected into a CogVideoX DiT via decoupled MLP embeddings added to the timestep embedding.
Key results
- MotionStone reaches 95.76% background consistency, 46.78% aesthetic quality and 62.29% imaging quality on WebVID, beating CogVideoX-5B (94.91%, 45.88%, 61.99%) and the other baselines.
- The motion estimator reaches 72.80% pairwise motion-relation accuracy versus 44.56% for SSIM, a gain of about 28 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MotionStone instead of CogVideoX-5B | Image-to-video generation · WebVID validation | Background Consistency: 94.91% -> 95.76% | Helped |
| use MotionStone instead of AnimateAnything | Image-to-video generation · WebVID validation | Aesthetic Quality: 46.04% -> 46.78% | Helped |
| use MotionStone instead of CogVideoX-5B | Image-to-video generation · WebVID validation | Imaging Quality: 61.99% -> 62.29% | Helped |
| use the decoupled motion estimator instead of SSIM | Motion estimation · Video-pair validation set | Motion Estimation Accuracy: 44.56% -> 72.80% | Helped |
| add the motion estimator rather than use fixed motion intensity (ablation) | MotionStone · WebVID validation | Imaging Quality: 60.15% -> 62.29% | Helped |
| replace feature-difference motion estimation with the decoupled motion estimator (ablation) | MotionStone · WebVID validation | Imaging Quality: 60.73% -> 62.29% | Helped |
| replace inter-frame SSIM motion estimation with the decoupled motion estimator (ablation) | MotionStone · WebVID validation | Imaging Quality: 54.75% -> 62.29% | Helped |
| use decoupled injection instead of mixing object and camera motion conditions (ablation) | MotionStone · WebVID validation | Imaging Quality: 58.73% -> 62.29% | Helped |
Surprising findings
- Traditional SSIM-based motion intensity injection lowered quality below the no-estimator baseline (92.99% background consistency), suggesting that poorly aligned motion signals can hurt training.
Limitations
- Motion estimator accuracy is only 72.80% on pair comparisons, so labels remain noisy.
- Training data is in-house and not released, limiting reproducibility.
- Evaluation uses VBench metrics that do not directly measure motion intensity control.
Experimental setup
Datasets: WebVID in-house 100,000 videos 5,000 human-annotated video pairs
Benchmarks: WebVID validation set
Metrics: VBench Background Consistency VBench Aesthetic Quality VBench Imaging Quality motion estimation accuracy vs. manual pair labels user study
Baselines: I2VGEN-XL SVD AnimateAnything CogVideoX-5B SSIM-based motion intensity (LivePhoto)
Builds on: CogVideoX TAdaConv VBench AnimateAnything
Closest papers by meaning
- Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think (CVPR 2025 · similarity 0.90)
- Decouple Content and Motion for Conditional Image-to-Video Generation (AAAI 2024 · similarity 0.87)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.86)
- MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation (NeurIPS 2025 · similarity 0.86)
- Move-in-2D: 2D-Conditioned Human Motion Generation (CVPR 2025 · similarity 0.86)
- Video Diffusion Models are Training-free Motion Interpreter and Controller (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.