Research map / Image generation / Fast diffusion sampling
FrameBridge: Improving Image-to-Video Generation with Bridge Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FrameBridge, which models image-to-video generation as a data-to-data bridge process from the replicated input frame (or a learned neural prior) instead of Gaussian noise, plus SNR-Aligned Fine-tuning to adapt pretrained T2V diffusion models.
Problem
Diffusion-based image-to-video models start from uninformative Gaussian noise, which is mismatched with the frame-to-frames nature of I2V and can hurt appearance consistency and synthesis quality.
Approach
Replace the Gaussian prior with a Dirac prior at the replicated image latent, building a denoising diffusion bridge from image to video; SAF reparameterizes and SNR-aligns the bridge input so pretrained diffusion T2V weights can be reused; a one-step neural prior network provides a coarse video estimate as a stronger prior.
Key results
- Zero-shot MSR-VTT FVD reduced from 192 (DynamiCrafter, 1.28M) to 95 for FrameBridge-VideoCrafter with SAF at 6.4M videos, versus 234 for DynamiCrafter at 6.4M.
- Non-zero-shot UCF-101 FVD improved from 171 (VDT-I2V) to 122 for FrameBridge-NP (neural prior).
- On VBench-I2V, FrameBridge-CogVideoX reaches total score 85.93, the highest among compared models (DynamiCrafter-256 84.35).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace diffusion generation with a replicated-frame bridge process | I2V generation on UCF-101 · UCF-101 non-zero-shot | FVD: 171 -> 154 | Helped |
| use neural prior instead of replicated-frame prior (ablation) | FrameBridge on UCF-101 · UCF-101 non-zero-shot | FVD: 154 -> 122 | Helped |
| add SAF to bridge fine-tuning (ablation) | FrameBridge fine-tuned from Latte-XL/2 · UCF-101 non-zero-shot, 5k iterations | FVD: 284 -> 141 | Helped |
| add SAF to bridge fine-tuning (ablation) | FrameBridge fine-tuned from Latte-XL/2 · UCF-101 non-zero-shot, 5k iterations | IS: 49.40 -> 55.98 | Helped |
| add SAF to bridge fine-tuning (ablation) | FrameBridge fine-tuned from Latte-XL/2 · UCF-101 non-zero-shot, 5k iterations | PIC: 0.6557 -> 0.8200 | Helped |
| add conditioning on the neural prior while retaining replicated prior (ablation) | FrameBridge · UCF-101 | FVD: 154 -> 129 | Helped |
| add conditioning on Fη(zᵢ,c) (ablation) | VDT-I2V · UCF-101 | FVD: 171 -> 132 | Helped |
| use SAF when fine-tuning from VideoCrafter1 (ablation) | FrameBridge-VideoCrafter · MSR-VTT zero-shot | FVD: 229 -> 99 | Helped |
| replace the diffusion process with FrameBridge using SAF | VideoCrafter1-based I2V · MSR-VTT zero-shot | FVD: 234 -> 95 | Helped |
| increase conditioning frame stride from 3 to 5 (ablation) | FrameBridge-FrameStride · VBench-I2V user study | AUR of dynamic degree: 2.74 -> 2.12 | Helped |
| increase conditioning frame stride from 3 to 5 (ablation) | FrameBridge-FrameStride · VBench-I2V user study | AUR of overall quality: 2.26 -> 2.34 | Mixed |
| add noise to the image condition during fine-tuning (ablation) | FrameBridge-NoisyCondition · VBench-I2V user study | AUR of dynamic degree: 2.74 -> 2.29 | Helped |
Surprising findings
- Neural prior alone (replicated prior, no neural prior) on UCF-101 ablation shows bridge without SAF matches diffusion at 10k iterations in FVD (176 vs 176), so gains depend on SAF at lower budgets.
Limitations
- Dynamic degree lower than some baselines (e.g., DynamiCrafter 38.69 vs FrameBridge-VideoCrafter 35.77), a trade-off with consistency.
- Neural prior is a simple one-step regression model; authors note more advanced priors could be designed.
- Experiments are limited to text/class-conditional settings on WebVid-2M and UCF-101 with moderate training budgets.
Experimental setup
Datasets: WebVid-2M UCF-101 MSR-VTT
Benchmarks: UCF-101 MSR-VTT VBench-I2V
Metrics: FVD IS PIC CLIPSIM VBench-I2V dimensions CD-FVD average user ranking
Baselines: DynamiCrafter SEINE I2VGen-XL SVD ConsistI2V SparseCtrl ExtDM VDT-I2V
Builds on: denoising diffusion bridge models (DDBM) latent video diffusion DynamiCrafter
Closest papers by meaning
- TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models (CVPR 2024 · similarity 0.89)
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis (CVPR 2025 · similarity 0.89)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.89)
- VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide (CVPR 2025 · similarity 0.88)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.88)
- Hierarchical Patch Diffusion Models for High-Resolution Video Generation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.