Research map / Video / Video generation
Rolling Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rolling Diffusion, a sliding-window diffusion process that assigns increasing noise levels to later frames in a sequence, enabling autoregressive video and fluid simulation generation with a local time reparameterization.
Problem
Standard diffusion models for temporal data treat all frames with equal noise, which is suboptimal for autoregressive or block-autoregressive generation of long sequences, causing high compute cost and error accumulation.
Approach
Reparameterize global diffusion time into a frame-dependent local time so that noise increases with frame index; denoise a sliding window of frames progressively as it shifts through the sequence, with an init schedule to handle boundary conditions.
Key results
- On BAIR Robot Pushing, rolling diffusion reaches FVD 59.6 vs 59.7 for standard diffusion, both beating prior methods such as Video Diffusion (66.9).
- On Kinetics-600 rollout with stride 8 (24 steps), rolling diffusion (beta=0.1) achieves FVD 39.8 vs 58.1 for standard diffusion.
- On Kolmogorov flow, rolling diffusion with init noise (2-8) reaches the best FSD at many time steps, e.g. FSD 27.68 at step 1 versus 39.59 for standard diffusion (2-1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use rolling diffusion instead of standard diffusion | Simple Diffusion · BAIR Robot Pushing | FVD: 59.7 -> 59.6 | No clear effect |
| use rolling diffusion instead of standard diffusion | Simple Diffusion · Kinetics-600 baseline (stride 1) | FVD: 3.9 -> 5.2 | Hurt |
| use rolling diffusion instead of standard diffusion with stride 8 | Simple Diffusion · Kinetics-600 rollout (stride=8, steps=24, Cond-Gen 5-11) | FVD: 58.1 -> 39.8 | Helped |
| use rolling diffusion instead of standard diffusion with stride 1 | Simple Diffusion · Kinetics-600 rollout (stride=1, steps=64, Cond-Gen 5-11) | FVD: 123.7 -> 211.2 | Hurt |
| use Rolling Diffusion instead of TECO | Kinetics-600 video prediction · Kinetics-600 (20 conditioning, 80 generated; 128×128) | FVD: 799 -> 685 | Helped |
| use rolling diffusion instead of standard diffusion | Simple Diffusion · Kolmogorov Flow rollout | FSD | Helped |
| increase the rolling window from 2-4 to 2-8 (ablation) | Rolling Diffusion (init noise) · Kolmogorov Flow @1 | FSD: 29.59 -> 27.68 | Helped |
| increase the rolling window from 2-4 to 2-8 (ablation) | Rolling Diffusion (init noise) · Kolmogorov Flow @8 | FSD: 61.93 -> 98.29 | Hurt |
| increase oversampling rate from β=0.1 to β=0.2 (ablation) | Rolling init rescaled · Kinetics-600 rollout (stride=8, steps=24, Cond-Gen 5-11) | FVD: 39.8 -> 44.1 | Hurt |
| increase oversampling rate from β=0.1 to β=0.5 (ablation) | Rolling init rescaled · Kinetics-600 rollout (stride=8, steps=24, Cond-Gen 5-11) | FVD: 39.8 -> 46.3 | Hurt |
| use the clipped init schedule with β=0.0 instead of rescaled init with β=0.1 (ablation) | Rolling Diffusion · Kinetics-600 rollout (stride=8, steps=24, Cond-Gen 5-11) | FVD: 39.8 -> 60.3 | Hurt |
| change oversampling rate from β=0.7 to β=0.9 (ablation) | Rolling init rescaled · Kinetics-600 rollout (stride=1, steps=64, Cond-Gen 5-11) | FVD: 216 -> 211 | Helped |
Surprising findings
- With stride 1 on Kinetics-600 (64 rollout steps), standard diffusion outperforms rolling diffusion (123.7 vs 211.2 FVD), which the authors attribute to low dynamics in the data.
- The MSE autoregressive model is optimal in MSE but performs poorly on distributional FSD metrics, showing MSE-trained models blur high-frequency content.
Limitations
- Rolling diffusion has no inherent runtime advantage over batch-autoregressive diffusion with similar parameter counts.
- Training may let the model over-rely on conditioning frames due to partial noising overlap, giving a weaker video prior.
- Underperforms standard diffusion on low-dynamics data (stride 1 Kinetics-600).
Experimental setup
Datasets: Kinetics-600 BAIR Robot Pushing Kolmogorov flow simulations (JaxCFD)
Benchmarks: BAIR Robot Pushing Kinetics-600 Kolmogorov flow
Metrics: FVD FSD MSE
Baselines: Standard Diffusion MSE autoregressive model Phenaki TrIVD-GAN-FP Video Diffusion RIN MAGVIT MAGVITv2 W.A.L.T.-L TECO DVD-GAN VideoGPT
Builds on: Simple Diffusion PDE Refiner variational diffusion models v-parameterization
Closest papers by meaning
- Elucidated Rolling Diffusion Models for Probabilistic Forecasting of Complex Dynamics (NeurIPS 2025 · similarity 0.90)
- ABC: Any-Subset Autoregression via Non-Markovian Diffusion Bridges in Continuous Time and Space (NeurIPS 2026 · similarity 0.88)
- Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Models (ICLR 2025 · similarity 0.87)
- Predicting partially observable dynamical systems via diffusion models with a multiscale inference scheme (NeurIPS 2025 · similarity 0.87)
- Non-stationary Diffusion For Probabilistic Time Series Forecasting (ICML 2025 · similarity 0.86)
- AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.