Research map / Image generation / Editing and style transfer
Re-Attentional Controllable Video Diffusion Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReAtCo, a training-free text-guided video editing framework that refocuses cross-attention maps toward user-specified object regions (RAD) and reinjects source invariant-region content during sampling (IRJS) to improve spatial controllability.
Problem
Text-guided video editing with T2I-based diffusion models lacks fine-grained controllability: edited objects are often mislocated and miscounted, and invariant background regions drift from the source due to accumulated sampling errors.
Approach
A Re-Attentional Diffusion (RAD) module applies inner- and outer-region top-k constraints on word-specific cross-attention maps using user-provided masks/boxes to align object placement, plus Invariant Region-guided Joint Sampling (IRJS) that replaces the background of each denoised latent with the diffused source background at every timestep, all without training.
Key results
- ReAtCo reaches VISOR 70.62 vs 25.31 for the next best (TCVE) and 15.62 for Tune-A-Video, with Frame Consistency 95.24 and Textual Alignment 28.64 (best among compared methods).
- Ablation: adding IRJS to RAD only raises VISOR from 69.06 to 70.62, while RAD alone lifts VISOR from 15.62 to 69.06.
- IRJS raises background reconstruction PSNR from 29.29 dB to 36.57 dB and lowers LPIPS from 0.2706 to 0.0206.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReAtCo instead of TCVE | Text-guided video editing · LOVEU-TGVE-2023 and evaluated video samples | VISOR: 25.31 -> 70.62 | Helped |
| Use ReAtCo instead of TCVE | Text-guided video editing · LOVEU-TGVE-2023 and evaluated video samples | Frame Consistency: 94.79 -> 95.24 | Helped |
| Use ReAtCo instead of TCVE | Text-guided video editing · LOVEU-TGVE-2023 and evaluated video samples | Textual Alignment: 27.71 -> 28.64 | Helped |
| Use ReAtCo instead of TokenFlow | Text-guided video editing · LOVEU-TGVE-2023 and evaluated video samples | PickScore: 20.57 -> 20.70 | Helped |
| Add IRJS (ablation) | ReAtCo with RAD · LOVEU-TGVE-2023 and evaluated video samples | Frame Consistency: 94.59 -> 95.24 | Helped |
| Add IRJS (ablation) | ReAtCo with RAD · LOVEU-TGVE-2023 and evaluated video samples | VISOR: 69.06 -> 70.62 | Helped |
| Add RAD (ablation) | ReAtCo with IRJS · LOVEU-TGVE-2023 and evaluated video samples | VISOR: 16.25 -> 70.62 | Helped |
| Add RAD (ablation) | ReAtCo with IRJS · LOVEU-TGVE-2023 and evaluated video samples | Textual Alignment: 26.94 -> 28.64 | Helped |
| Add IRJS (ablation) | Tune-A-Video-based ReAtCo · Invariant region | PSNR (dB): 29.29 -> 36.57 | Helped |
| Add IRJS (ablation) | Tune-A-Video-based ReAtCo · Invariant region | LPIPS: 0.2706 -> 0.0206 | Helped |
| Set K to 20% in top_k (ablation) | RAD · LOVEU-TGVE-2023 and evaluated video samples | VISOR | Helped |
Surprising findings
- Removing IRJS only slightly lowers VISOR, while RAD accounts for most of the spatial gain.
- Applying the attention constraint to all cross-attention responses degrades editing; restricting it to the top 20% of elements works best.
Limitations
- Temporal unsmoothness inherited from the Tune-A-Video backbone.
- Gradient tracking through the whole model increases GPU memory, limiting frames processed at once.
- Requires user-specified object regions (boxes or masks) and target words.
Experimental setup
Datasets: LOVEU-TGVE-2023 video samples from prior work
Benchmarks: LOVEU-TGVE-2023
Metrics: Frame Consistency Textual Alignment PickScore VISOR PSNR LPIPS
Baselines: Tune-A-Video FateZero StableVideo TokenFlow TCVE
Builds on: Tune-A-Video DDIM inversion Stable Diffusion FateZero TokenFlow
Closest papers by meaning
- VideoDirector: Precise Video Editing via Text-to-Video Models (CVPR 2025 · similarity 0.88)
- FADE: Frequency-Aware Diffusion Model Factorization for Video Editing (CVPR 2025 · similarity 0.88)
- VideoGrain: Modulating Space-Time Attention for Multi-Grained Video Editing (ICLR 2025 · similarity 0.87)
- FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing (ICLR 2024 · similarity 0.87)
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.87)
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.