Research map / Video / Video generation
Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fairy adapts an instruction-based image-editing diffusion model to video via anchor-based cross-frame attention (parallelizable across GPUs) plus equivariant finetuning with affine augmentation, achieving fast, temporally consistent instruction-guided video editing.
Problem
Instruction-guided video-to-video editing with image-editing diffusion models suffers from temporal inconsistency when frames are edited independently, and prior tracking-and-propagation or video-model methods are fragile, memory-heavy, or slow.
Approach
Replace self-attention with cross-frame attention over cached key/value features of a small set of uniformly sampled anchor frames, so each frame is edited independently given anchors (enabling multi-GPU parallelism and caching), and finetune the image editing model with affine-transformation data augmentation applied jointly to input and edited images to make it equivariant to affine transforms.
Key results
- Fairy generates 120-frame 512x384 videos in about 14 seconds on 8 A100 GPUs, reported as 44x faster than Rerender and 53x faster than TokenFlow.
- In A/B human study, Fairy beats TokenFlow 73% to 16%, Gen-1 72% to 26%, and Rerender 41% to 36%.
- Ablation Tem-Con rises from 0.959 (frame-by-frame) to 0.968 (with anchors) to 0.974 (anchors plus equivariant finetuning).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Fairy instead of TokenFlow | Video editing · 4-second, 30 FPS, 512p×384p video | Latency (sec): 744 -> 13.8 | Helped |
| use Fairy instead of Rerender | Video editing · 4-second, 30 FPS, 512p×384p video | Latency (sec): 608 -> 13.8 | Helped |
| use Fairy instead of TokenFlow | Video editing · Video editing | Frame-Acc: 0.537 -> 0.819 | Helped |
| use Fairy instead of Rerender | Video editing · Video editing | Frame-Acc: 0.775 -> 0.819 | Helped |
| use Fairy instead of TokenFlow | Video editing · Video editing | Tem-Con: 0.973 -> 0.974 | Helped |
| use Fairy instead of Rerender | Video editing · Video editing | Tem-Con: 0.972 -> 0.974 | Helped |
| use Fairy rather than Rerender | Video editing · A/B comparison vs. Rerender | Overall quality preference for Fairy: 41% vs 36% | Helped |
| use Fairy rather than TokenFlow | Video editing · A/B comparison vs. TokenFlow | Overall quality preference for Fairy: 73% vs. 16% | Helped |
| use Fairy rather than Gen-1 | Video editing · A/B comparison vs. Gen-1 | Overall quality preference for Fairy: 72% vs 26% | Helped |
| add anchor-based attention (ablation) | Fairy · Video editing | Tem-Con: 0.959 -> 0.968 | Helped |
| add equivariant fine-tuning (ablation) | Fairy with anchors · Video editing | Tem-Con: 0.968 -> 0.974 | Helped |
Surprising findings
- Cross-frame attention scores alone (without finetuning) track points on DAVIS with over 60%/70% accuracy at delta-x 16/32 in early and late UNet layers.
Limitations
- Quality is bounded by the underlying image-editing model, including face and text distortion.
- Equivariant finetuning makes dynamic effects such as lightning or flames appear static.
- Baseline results were collected from a prior source rather than rerun, and Gen-1 was evaluated on a smaller 100-video set.
Experimental setup
Datasets: image editing dataset used to train the base image-editing model (unnamed) ShutterStock videos (evaluation) DAVIS (TAP-Vid, tracking analysis)
Benchmarks: DAVIS (TAP-Vid) position accuracy user study on 1000 video-instruction samples
Metrics: A/B human preference win rate Tem-Con (CLIP temporal consistency) Frame-Acc latency (sec) position accuracy delta-x
Baselines: Rerender TokenFlow Gen-1
Builds on: instruction-based image editing diffusion model (InstructPix2Pix-style) cross-frame attention TokenFlow Rerender
Closest papers by meaning
- Transforming Image Editors into Video Editors (NeurIPS 2026 · similarity 0.89)
- BIVDiff: A Training-Free Framework for General-Purpose Video Synthesis via Bridging Image and Video Diffusion Models (CVPR 2024 · similarity 0.88)
- Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation (ICLR 2025 · similarity 0.87)
- Training-Free Efficient Video Generation via Dynamic Token Carving (NeurIPS 2025 · similarity 0.87)
- COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing (NeurIPS 2024 · similarity 0.87)
- Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.