Research map / Video / Video generation
Generative Image Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns an image-space prior over natural oscillatory scene motion by having a frequency-coordinated latent diffusion model predict spectral volumes from a single image, which are rendered into videos via an image-based rendering module.
Problem
Generating realistic, long-term oscillatory scene motion (trees, flowers, candles, clothes swaying) from a single still image, with temporal coherence over entire videos.
Approach
Represent dense long-term pixel motion as Fourier-domain spectral volumes and generate them with a latent diffusion model whose frequency slices are coordinated by attention layers, plus frequency-adaptive normalization; the motion then drives softmax-splatting image-based rendering.
Key results
- Achieves FVD 47.1 and DTFVD 2.53 versus 166.0 and 5.35 for the best prior baseline (Endo et al.) on the test set.
- Users prefer its animations over AnimateDiff, ModelScope, and Gen-2 with 80.9% preference in a 30-video study.
- Ablations show frequency-coordinated denoising and adaptive normalization each improve FVD and DTFVD over simpler variants (e.g., full FVD 47.1 vs 52.5 for independent prediction).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed spectral-volume motion prediction and rendering method instead of Endo et al. | Single-image animation methods · Test set | FVD: 166.0 -> 47.1 | Helped |
| use the proposed method instead of Endo et al. | Single-image animation methods · Test set | FID: 10.4 -> 4.03 | Helped |
| use K = 16 frequency bands rather than K = 4 (ablation) | Generative Image Dynamics · Test set | FVD: 60.3 -> 47.1 | Helped |
| use K = 16 frequency bands rather than K = 8 (ablation) | Generative Image Dynamics · Test set | FVD: 52.1 -> 47.1 | Helped |
| use K = 16 frequency bands rather than K = 24 (ablation) | Generative Image Dynamics · Test set | FVD: 48.2 -> 47.1 | Helped |
| add frequency adaptive normalization (ablation) | Generative Image Dynamics · Test set | FVD: 62.7 -> 47.1 | Helped |
| add frequency-coordinated denoising rather than independent frequency prediction (ablation) | Generative Image Dynamics · Test set | FVD: 52.5 -> 47.1 | Helped |
| use frequency-coordinated denoising rather than joint 4K-channel volume prediction (ablation) | Generative Image Dynamics · Test set | FVD: 53.7 -> 47.1 | Helped |
| use the full rendering module rather than single-scale softmax splatting with learnable weights (ablation) | Generative Image Dynamics · Test set | FVD: 49.5 -> 47.1 | Helped |
| generate motion rather than repeat the input image (ablation) | Generative Image Dynamics · Test set | FVD: 237.5 -> 47.1 | Helped |
Surprising findings
- Off-the-shelf large video diffusion models (AnimateDiff, ModelScope, Gen-2) often fail to adhere to the input image or drift in color and distortion over time.
Limitations
- Only predicts low-frequency components, so it fails to model non-oscillating motion or high-frequency vibrations.
- Artifacts on thin moving objects or large displacements where motion trajectories are unreliable.
- Degradation when motions require generating large amounts of unseen content.
Experimental setup
Datasets: 3,015 online and self-captured videos of natural oscillatory scenes Dynamic Textures Database Kinetics
Benchmarks: held-out test set of Generative Image Dynamics videos
Metrics: FID KID FVD FVD32 DTFVD DTFVD32 sliding window FID sliding window DTFVD user preference
Baselines: TATS Stochastic I2V MCVD LFDM DMVFN Endo et al. Holynski et al. AnimateDiff ModelScope Gen-2
Builds on: Latent Diffusion Models spectral volumes (Davis et al.) softmax splatting classifier-free guidance universal guidance
Closest papers by meaning
- Spectral Progressive Diffusion for Efficient Image and Video Generation (NeurIPS 2026 · similarity 0.88)
- Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation (ICLR 2025 · similarity 0.87)
- Generative Modeling with Phase Stochastic Bridge (ICLR 2024 · similarity 0.87)
- MotionCraft: Physics-Based Zero-Shot Video Generation (NeurIPS 2024 · similarity 0.87)
- VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models (ICML 2025 · similarity 0.87)
- Human Motion Diffusion as a Generative Prior (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.