Research map / Vision-language and multimodal / Human and embodied modeling
Animating General Image with Large Visual Motion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LVMM, a latent motion diffusion model with a motion VAE and neural image renderer that animates a single still image into a video of plausible local motion (e.g., swaying vegetation, facial expressions).
Problem
Animating a single generic image with realistic local dynamics (wind-driven motion, facial expressions) without an explicit video generation prior, where motion is ambiguous and mixed with visual appearance.
Approach
Decomposes the latent motion vector into a visual component (v = z0 from the reference image) and a motion component (u = zk - v0), then trains a 3D U-Net latent diffusion model only on the motion component, conditioned on ImageBind features of the reference image; a neural image renderer warps feature pyramids using optical flow and an intention map.
Key results
- On MUG, LVMM (fine-tuned from LVMM-Facial) reaches FVD 15.34 versus 27.57 for LFDM.
- On MHAD, LVMM trained from scratch reaches FVD 143.67 versus 152.48 for LFDM; fine-tuned variant reaches 124.56.
- On NATOPS, LVMM fine-tuned reaches FVD 144.46 versus 160.84 for LFDM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LVMM instead of LFDM | MUG video generation · MUG | FVD: 27.57 -> 15.34 | Helped |
| Use LVMM instead of LFDM | MHAD video generation · MHAD | FVD: 152.48 -> 124.56 | Helped |
| Use LVMM instead of LFDM | NATOPS video generation · NATOPS | FVD: 160.84 -> 144.46 | Helped |
| Add fine-tuning of the Neural Image Renderer after VAE reconstruction (ablation) | LVMM motion rendering model · Facial expression reconstruction | visual detail quality | Helped |
| Predict only optical flow δx,y, without the intention map or neural image rendering (ablation) | LVMM motion rendering model · Facial expression reconstruction | image sharpness | Hurt |
| Use the intention map with direct warp-function rendering instead of the neural image renderer (ablation) | LVMM motion rendering model · Facial expression reconstruction | motion complexity | Hurt |
| Render warped features with the neural image renderer (ablation) | LVMM motion rendering model · Facial expression reconstruction | motion complexity | Helped |
| Fine-tune LVMM-Facial weights rather than train the model from scratch | LVMM on MUG · MUG | FVD: 23.47 -> 15.34 | Helped |
| Fine-tune LVMM-Facial weights rather than train the model from scratch | LVMM on MHAD · MHAD | FVD: 143.67 -> 124.56 | Helped |
| Fine-tune LVMM-Facial weights rather than train the model from scratch | LVMM on NATOPS · NATOPS | FVD: 157.92 -> 144.46 | Helped |
Surprising findings
- The latent motion vector is found to entangle visual appearance with motion, so predicting motion flow directly is harder than predicting the decomposed motion component.
Limitations
- The generalized motion diffusion model cannot yet produce high-quality motion trajectories for arbitrary motion forms zero-shot, per the authors.
- Evaluation is limited to small benchmarks (MUG, MHAD, NATOPS) and visual comparisons with closed-source Gen-2 and I2V-GenXL.
- Qualitative comparison only with I2V-GenXL and Gen-2 because their training data is undisclosed.
Experimental setup
Datasets: WebVid10M HDVILA-100M CelebV-HD MUG MHAD NATOPS SHM FMM
Benchmarks: MUG MHAD NATOPS
Metrics: FVD cFVD sFVD
Baselines: ImaGINator VDM LDM LFDM I2V-GenXL Gen-2
Builds on: DDPM LFDM ImageBind Siarohin et al. first-order motion model
Closest papers by meaning
- VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models (ICML 2025 · similarity 0.85)
- Scaling Large Motion Models with Million-Level Human Motions (ICML 2025 · similarity 0.85)
- Generative Image Dynamics (CVPR 2024 · similarity 0.85)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.85)
- Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (ICLR 2024 · similarity 0.85)
- LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.