Research map / Video / Video generation
FADE: Frequency-Aware Diffusion Model Factorization for Video Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FADE, a training-free video editing method that factorizes a pre-trained text-to-video DiT into frequency-based sketching and sharpening blocks and uses low-pass 3D-DFT spectrum-guided modulation of the sampling trajectory.
Problem
Video editing with diffusion models is hard: image-diffusion methods fail on temporal/motion edits, and adapting large text-to-video models is memory- and compute-heavy, while null-text optimization and per-video tuning are slow.
Approach
Analyzes attention patterns across transformer blocks of a T2V model to find early "sketching" blocks (low-frequency layout and motion) versus later "sharpening" blocks (high-frequency detail), then uses only the sketching blocks' full-attention outputs, low-pass filtered in the 3D Fourier domain, to compute a guidance gradient that modulates DDIM sampling instead of attention injection.
Key results
- On 20 videos, FADE reaches CLIP 0.3762 for appearance edits and 0.3683 for motion edits, the highest CLIP in both settings versus CogVideoX-V2V (0.3754, 0.3678).
- FADE achieves the lowest OSV on appearance edits (31.36 vs 31.45 for CogVideoX-V2V) and 32.28 on motion edits, while Mask PSNR is 20.69 (appearance) and 19.26 (motion).
- Editing with the sketching blocks only takes about 3 minutes per video versus over 15 minutes for previous methods, according to the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FADE instead of CogVideoX-V2V | Video editing methods · Appearance edits | CLIP: 0.3754 -> 0.3762 | Helped |
| use FADE instead of CogVideoX-V2V | Video editing methods · Motion edits | OSV: 35.52 -> 32.28 | Helped |
| use FADE instead of the strongest baseline | Video editing methods · Appearance edits | OSV: 31.45 -> 31.36 | Helped |
| use FADE instead of the strongest baseline | Video editing methods · Motion edits | CLIP: 0.3678 -> 0.3683 | Helped |
| use only sketching blocks rather than symmetric combinations of sketching and sharpening blocks (ablation) | FADE · Video editing benchmark | CLIP: 0.3659 -> 0.3728 | Helped |
| factorize the model and use sketching blocks rather than all blocks (ablation) | FADE · Video editing benchmark | CLIP: 0.3691 -> 0.3728 | Helped |
| add a low-pass filter to spectrum-guided modulation (ablation) | FADE · Video editing benchmark | CLIP: 0.3612 -> 0.3728 | Helped |
Surprising findings
- Including the later sharpening blocks in the guidance degraded editing quality rather than improving it, which the authors attribute to misdirected attention.
- Using the full unfiltered spectrum in the guidance weakens text-video alignment, because high-frequency source content leaks into the edit.
Limitations
- Performance depends on the underlying video model.
- Struggles with heavy occlusions requiring advanced temporal reasoning.
- Evaluated on only 20 videos for quantitative results.
Experimental setup
Datasets: DAVIS
Benchmarks: DAVIS
Metrics: CLIP score OSV Mask PSNR LPIPS user preference
Baselines: Tune-A-Video Video-P2P FateZero CogVideoX-V2V
Builds on: DDIM inversion Video-P2P FateZero CogVideoX
Closest papers by meaning
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.89)
- MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance (AAAI 2024 · similarity 0.89)
- MotionEditor: Editing Video Motion via Content-Aware Diffusion (CVPR 2024 · similarity 0.89)
- Ouroboros-Diffusion: Exploring Consistent Content Generation in Tuning-free Long Video Diffusion (AAAI 2025 · similarity 0.89)
- RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models (CVPR 2024 · similarity 0.89)
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.