Research map / Video / Video generation
How I Warped Your Noise: a Temporally-Correlated Noise Prior for Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes int-noise, a continuous white-noise interpretation of diffusion noise with a distribution-preserving transport equation, enabling temporally correlated noise warping for diffusion-based video tasks.
Problem
Image-based diffusion models used for video editing/generation produce flickering or texture-sticking artifacts because standard noise sampling (random or fixed) does not preserve correlations between frames, and naive warping of noise (interpolation) breaks its distributional properties.
Approach
Reinterpret discrete Gaussian noise pixels as integrals of an underlying infinite-resolution white noise field, and warp it with a noise transport equation that includes a Jacobian-determinant variance rescaling, so warped noise stays unit-variance and temporally correlated.
Key results
- On appearance transfer, int-noise achieves warp error 2.50e-3 versus 10.00e-3 for Random Noise and 4.35e-3 for Fixed Noise, with FID 92.63 comparable to Fixed/PYoCo rather than interpolation methods (FID above 140).
- On 4x video super-resolution, int-noise reaches warp error 6.49e-3 versus 8.91e-3 for Random Noise, with LPIPS 0.196 close to Random's 0.192.
- Bilinear interpolation gives lowest warp error on pose-to-person (2.13) but with much worse visual quality (FID 143 vs 92.63 for int-noise on appearance transfer), while int-noise is less computationally efficient than baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ∫-noise instead of random noise | SDEdit · Appearance Transfer | warp (× 10^-3): 10.00 -> 2.50 | Helped |
| Use ∫-noise instead of random noise | SDEdit · Appearance Transfer | FID: 74.75 -> 92.63 | Hurt |
| Use ∫-noise instead of random noise | SDEdit · Appearance Transfer | Precision: 0.719 -> 0.661 | Hurt |
| Use ∫-noise instead of random noise | I²SB · Video SR (4×) | warp: 8.91 -> 6.49 | Helped |
| Use ∫-noise instead of random noise | I²SB · Video SR (4×) | LPIPS: 0.192 -> 0.196 | Hurt |
| Use ∫-noise instead of random noise | I²SB · JPEG restore | warp: 8.28 -> 5.98 | Helped |
| Use ∫-noise instead of random noise | I²SB · JPEG restore | LPIPS: 0.163 -> 0.165 | Hurt |
| Use ∫-noise instead of random noise | PIDM · Pose-to-Person | warp: 34.22 -> 2.92 | Helped |
| Use ∫-noise instead of fixed noise | PIDM · Pose-to-Person | warp: 2.26 -> 2.92 | Hurt |
| Replace ∫-noise with bilinear interpolation (ablation) | Noise warping · Appearance Transfer | noise variance | Hurt |
| Replace ∫-noise with nearest-neighbor interpolation (ablation) | Noise warping · Appearance Transfer | pixel independence | Hurt |
| Use a lower noise upsampling factor k (ablation) | ∫-noise warping · Fluid super-resolution | undefined warped-noise pixels | Hurt |
Surprising findings
- Noise warping has limited impact on temporal coherency in latent diffusion models because latent noise controls mostly low-frequency structure.
- On pose-to-person, the quantitative warp error disagrees with visual assessment, with the authors analysing this discrepancy in the appendix.
Limitations
- Noise warping algorithm is computationally more expensive than simpler interpolation-based correlation methods.
- Temporal coherency gains depend on the diffusion pipeline and training data; constrained pipelines are less sensitive to the noise prior.
- Limited effect in latent diffusion models due to lower-resolution latent noise.
Experimental setup
Datasets: LSUN Bedroom 2D fluid simulation dataset
Benchmarks: LSUN Bedroom appearance transfer Video super-resolution (4x) with I2SB JPEG restoration with I2SB PIDM pose-to-person Fluid super-resolution
Metrics: warping error FID Improved Precision LPIPS
Baselines: Random Noise Fixed Noise PYoCo (mixed) PYoCo (prog.) Control-A-Video Bilinear interpolation Bicubic interpolation Nearest neighbor interpolation
Builds on: SDEdit I2SB PIDM DDIM inversion Control-A-Video
Closest papers by meaning
- Infinite-Resolution Integral Noise Warping for Diffusion Models (ICLR 2025 · similarity 0.91)
- FreqPrior: Improving Video Diffusion Models with Frequency Filtering Gaussian Noise (ICLR 2025 · similarity 0.89)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.89)
- Spectral Progressive Diffusion for Efficient Image and Video Generation (NeurIPS 2026 · similarity 0.89)
- NoiseDiffusion: Correcting Noise for Image Interpolation with Diffusion Models beyond Spherical Linear Interpolation (ICLR 2024 · similarity 0.88)
- Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.