Research map / Video / Video generation
FreeMask: Rethinking the Importance of Attention Masks for Zero-Shot Video Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FreeMask, which selects cross-attention masks per model layer and denoising timestep using a Mask Matching Cost (MMC) against DAVIS ground truth, then applies masked blending across temporal, cross, and self-attention for zero-shot video editing.
Problem
Naively using cross-attention masks for zero-shot video editing causes blurring and flickering, because mask precision varies across U-Net layers, denoising timesteps, and T2V models.
Approach
Quantify mask precision with a MIoU-based Mask Matching Cost (LMMC across layers, TMMC across timesteps), select time-agnostic or time-aware masks adaptively by whether the edit changes the object word, and blend masks into temporal, cross, and self-attention features to decouple unedited regions without tuning blend ratios.
Key results
- On the quantitative comparison table, FreeMask reaches Temp 0.952, CLIP 0.282, M.PSNR 25.94 and LPIPS 0.366, the best Temp, M.PSNR and LPIPS among compared methods.
- In the user study, FreeMask scores 7.68 for text alignment, 9.23 for image fidelity and 8.90 for quality, the highest on all three.
- Applying MMC-selected masks to FateZero improves its temporal consistency and editing quality, per the paper's extension experiment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TMMC-based time-aware mask selection for shape editing (ablation) | FreeMask · Jeep-to-Porsche shape editing | Editing quality | Helped |
| Add LMMC-based layer selection (ablation) | FreeMask · Jeep-to-Porsche shape editing | Structural detail preservation | Helped |
| Use a time-agnostic mask instead of a time-aware mask (ablation) | FreeMask · Stylization | Background structure preservation | Helped |
| Add masked latent blending to masked attention blending (ablation) | FreeMask · Stylization | Detail coherence | Helped |
| Use excessive latent blending (ablation) | FreeMask · Video editing | Editability | Hurt |
| Add masked self-attention blending to self-attention blending (ablation) | FreeMask · Jeep-to-Porsche shape editing | Structural detail preservation | Helped |
| Add masked temporal-attention blending (ablation) | FreeMask · Video editing | Temporal detail preservation | Helped |
| Add masked cross-attention blending (ablation) | FreeMask · Video editing | Boundary preservation | Helped |
| Add MMC-selected masks to FateZero | FateZero · Video editing | Temporal consistency | Helped |
| Add MMC-selected masks to FateZero | FateZero · Video editing | Image editing quality | Helped |
Surprising findings
- Cross-attention mask precision is not constant: it peaks in middle layers (inverted U-shape across layers) and improves with denoising progress, and this pattern is systematic per model architecture.
Limitations
- Struggles with multi-object scenes where objects move along different trajectories.
- Focuses on appearance editing and does not address motion editing.
- Relies on the base T2V model's capacity for complex motion patterns.
Experimental setup
Datasets: DAVIS
Benchmarks: DAVIS Pexels internet videos
Metrics: Temp (CLIP frame consistency) CLIP score Masked PSNR LPIPS user study (Edit, Image, Quality)
Baselines: FateZero TokenFlow ControlVideo AnyV2V Tune-A-Video Video-P2P Text2Video-Zero Pix2Video
Builds on: Prompt-to-Prompt DDIM inversion FateZero Tune-A-Video
Closest papers by meaning
- MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers (CVPR 2024 · similarity 0.87)
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.86)
- VideoGrain: Modulating Space-Time Attention for Multi-Grained Video Editing (ICLR 2025 · similarity 0.86)
- Click2Mask: Local Editing with Dynamic Mask Generation (AAAI 2025 · similarity 0.86)
- VidToMe: Video Token Merging for Zero-Shot Video Editing (CVPR 2024 · similarity 0.86)
- FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.