Research map / Video / Video generation
FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free text-to-video editing that adds optical-flow-guided attention (FLATTEN) over patch trajectories inside a T2I diffusion U-Net to improve cross-frame consistency.
Problem
Text-to-video editing with image diffusion models suffers from visual inconsistency across frames because dense spatio-temporal attention aggregates irrelevant patches.
Approach
Use RAFT optical flow to sample patch trajectories in latent space and restrict attention so each patch attends only to patches on the same flow path across frames, applied on top of dense spatio-temporal attention without new parameters or training.
Key results
- On TGVE-D, FLATTEN reaches S_edit 57.01 vs 50.21 for TokenFlow (the next best), with CLIP-T 28.05 and E_warp 4.92.
- Plugging FLATTEN into ControlVideo on TGVE-D reduces E_warp from 6.81 to 4.78 and raises S_edit from 40.70 to 56.42.
- In a 16-participant user study, FLATTEN received the highest preference in semantic (31.46), consistency (41.12) and motion (41.59) ratings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add flow-guided attention to ControlVideo | ControlVideo · TGVE-D | S_edit: 40.70 to 56.42 | Helped |
| Add dense spatio-temporal attention (ablation) | Base · TGVE-D | S_edit: 21.16 -> 42.06 | Helped |
| Add flow-guided attention (ablation) | Base · TGVE-D | S_edit: 21.16 -> 44.69 | Helped |
| Add DSTA and FLATTEN, recomputing FLATTEN queries, keys, and values with linear projections (ablation) | Base · TGVE-D | S_edit: 21.16 -> 49.93 | Helped |
| Use DSTA output directly as FLATTEN queries, keys, and values (ablation) | Base + DSTA + FLATTEN · TGVE-D | S_edit: 49.93 -> 57.01 | Helped |
| Use DSTA output directly rather than recomputing FLATTEN queries, keys, and values with linear projections (ablation) | Base + DSTA + FLATTEN · TGVE-D | CLIP-T: 27.96 -> 28.05 | No clear effect |
| Use FLATTEN during DDIM inversion | FLATTEN reconstruction · Source video reconstruction | PSNR: 32.74dB -> 33.89dB | Helped |
| Use FLATTEN during DDIM inversion | FLATTEN reconstruction · Source video reconstruction | SSIM: 0.8974 -> 0.9159 | Helped |
| Use flow estimated by GMA instead of RAFT | FLATTEN · Video editing | visual output difference | No clear effect |
| Add random Gaussian noise to the pre-computed optical flow | FLATTEN · Video editing | editing quality | Hurt |
| Use optical-flow guidance instead of the baseline without optical flow | FLATTEN · Video editing | editing quality | Helped |
Surprising findings
- Applying FLATTEN in DDIM inversion as well as sampling improves reconstruction (PSNR 33.89 dB vs 32.74 dB without).
- Feeding DSTA output directly as queries/keys/values for FLATTEN worked better than recomputing them with new linear projections, which sometimes caused blurring.
Limitations
- Limited for dramatic structure editing (e.g., sharks turned into drones fail).
- Trajectory sampling relies on optical flow quality; occlusions are handled by randomly choosing among conflicting trajectories.
- Warping error is only slightly better than TokenFlow (0.1e-3 lower on TGVE-V is actually higher), and the S_edit metric is a simple ratio of CLIP-T to E_warp.
Experimental setup
Datasets: LOVEU-TGVE DAVIS Videvo
Benchmarks: LOVEU-TGVE (TGVE-D and TGVE-V subsets)
Metrics: CLIP-F PickScore CLIP-T E_warp S_edit user preference
Baselines: Tune-A-Video Text2Video-Zero ControlVideo FateZero TokenFlow
Builds on: DDIM inversion Tune-A-Video RAFT Prompt-to-Prompt PnP-Diffusion
Closest papers by meaning
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024 · similarity 0.89)
- VideoGrain: Modulating Space-Time Attention for Multi-Grained Video Editing (ICLR 2025 · similarity 0.89)
- FADE: Frequency-Aware Diffusion Model Factorization for Video Editing (CVPR 2025 · similarity 0.88)
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.88)
- VideoDirector: Precise Video Editing via Text-to-Video Models (CVPR 2025 · similarity 0.87)
- SteerVTE: Seamless Video Text Editing with Style and Glyph Control (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.