Research map / Video / Video generation
FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FreqPrior, a frequency-filtering noise-prior refinement for text-to-video diffusion that preserves Gaussian variance (using squared low/high-pass complementarity) and uses partial sampling to cut inference time, improving VBench scores over FreeInit.
Problem
Existing frequency-based noise priors for video diffusion (FreeInit) suffer variance decay, causing videos with reduced motion dynamics and detail loss, and require costly full sampling iterations.
Approach
Refines initial noise in the frequency domain using a low-pass filter M combined with sqrt(1-M^2) for the high-pass part, so the sum of squared filters equals one and the refined noise remains close to standard Gaussian; also perturbs the latent at an intermediate timestep (t=321) to shorten the prior-finding sampling.
Key results
- On VBench, FreqPrior achieves total score 68.77 on VideoCrafter vs 68.29 for FreeInit and 67.42 for tripled-step Gaussian, with inference time 63.67s vs 83.18s.
- On ModelScope, total score 73.04 vs 72.34 for FreeInit, with inference time 44.88s vs 57.73s.
- On AnimateDiff, total score 78.11 vs 77.45 for Gaussian and 77.43 for FreeInit; numerical covariance error is ~1e-26 vs 3.8-17 for FreeInit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FreqPrior instead of FreeInit | VideoCrafter · VBench | quality score: 70.62 -> 70.63 | Helped |
| Use FreqPrior instead of FreeInit | VideoCrafter · VBench | semantic score: 58.97 -> 61.33 | Helped |
| Use FreqPrior instead of FreeInit | VideoCrafter · VBench | total score: 68.29 -> 68.77 | Helped |
| Use FreqPrior instead of FreeInit | ModelScope · VBench | total score: 72.34 -> 73.04 | Helped |
| Use FreqPrior instead of FreeInit | AnimateDiff · VBench | semantic score: 68.85 -> 70.37 | Helped |
| Use partial sampling rather than full sampling to find the prior | FreqPrior vs FreeInit · VBench | inference time: 83.18s -> 63.67s; 57.73s -> 44.88s; 70.45s -> 54.05s | Helped |
| Vary cosθ across 1.0, 0.9, 0.8, and 0.7 (ablation) | FreqPrior · VBench total score | total score: 69.02, 68.96, 68.91, 69.04; 72.82, 72.92, 73.12, 72.97; 78.07, 78.07, 78.12, 78.09 | No clear effect |
| Increase diffusion timestep t (ablation) | FreqPrior · VBench | total score | Hurt |
Surprising findings
- Mixed and progressive PYoCo noise priors crash pretrained video diffusion models without dedicated training.
- FreeInit reduces semantic and total score relative to Gaussian noise on AnimateDiff.
Limitations
- Quality reported only via VBench; total score gains over FreeInit are small (about 0.3-0.5 points).
- Performance decreases slowly as intermediate timestep t increases, creating a quality-speed trade-off.
- Analysis relies on the assumption that the post-diffusion noise is standard Gaussian.
Experimental setup
Datasets: VBench
Benchmarks: VBench
Metrics: quality score semantic score total score inference time covariance error
Baselines: Gaussian noise mixed noise prior progressive noise prior FreeInit tripled-step Gaussian
Builds on: FreeInit PYoCo DDIM
Closest papers by meaning
- Spectral Progressive Diffusion for Efficient Image and Video Generation (NeurIPS 2026 · similarity 0.90)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.90)
- FreqTS: Frequency-Aware Token Selection for Accelerating Diffusion Models (AAAI 2025 · similarity 0.89)
- How I Warped Your Noise: a Temporally-Correlated Noise Prior for Diffusion Models (ICLR 2024 · similarity 0.89)
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.89)
- Boosting Diffusion Models with Moving Average Sampling in Frequency Domain (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.