Research map / Image generation / Text-to-image diffusion
MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MaskINT, a two-stage text-based video editing pipeline that jointly edits two keyframes with a zero-shot T2I ControlNet and then fills in intermediate frames with a structure-aware non-autoregressive masked transformer trained on video-only data.
Problem
Text-based video editing with diffusion models is computationally expensive and often requires large paired text-video datasets, and extended-attention zero-shot methods are slow because they denoise every frame.
Approach
Edit only keyframes (first and last) with a pre-trained T2I model using extended self-attention, then use a masked generative transformer conditioned on HED structure maps and the edited endpoint frames to interpolate all intermediate frames in parallel via MaskGiT-style decoding.
Key results
- MaskINT reaches 0.311 DAVIS prompt consistency and 0.952 temporal consistency, comparable to diffusion baselines, while running in 22s per 16-frame clip versus 120s for ControlVideo and 150s for TokenFlow.
- On frame interpolation with original keyframes, MaskINT attains 22.15 PSNR on DAVIS versus 17.00 for FILM and 17.31 for RIFE.
- The paper reports up to 5-7x faster inference than diffusion-based video editing baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace TokenFlow with MaskINT for video editing | video editing · 16-frame video clip | inference time: 150s -> 22s | Helped |
| use structure-aware MaskINT interpolation instead of FILM | video frame interpolation · DAVIS | PSNR: 17.00 -> 22.15 | Helped |
| use structure-aware MaskINT interpolation instead of FILM | video frame interpolation · ShutterStock | PSNR: 20.90 -> 24.19 | Helped |
| increase keyframes from 1 to 2 (ablation) | MaskINT · video editing | temporal consistency: 0.9690 -> 0.9714 | Helped |
| increase keyframes from 2 to 3 (ablation) | MaskINT · video editing | temporal consistency: 0.9714 -> 0.9721 | Helped |
| increase keyframes from 3 to 4 (ablation) | MaskINT · video editing | temporal consistency: 0.9721 -> 0.9728 | Helped |
| increase keyframes from 4 to 6 (ablation) | MaskINT · video editing | temporal consistency: 0.9728 -> 0.9737 | Helped |
| increase decoding steps from 16 to 32 (ablation) | MaskINT · video editing | temporal consistency: 0.9691 -> 0.9714 | Helped |
| increase decoding steps from 32 to 64 (ablation) | MaskINT · video editing | temporal consistency: 0.9714 -> 0.9719 | Helped |
| increase decoding steps from 64 to 128 (ablation) | MaskINT · video editing | temporal consistency: 0.9719 -> 0.9720 | Helped |
Surprising findings
- Increasing the number of jointly edited keyframes improves prompt and temporal consistency but increases runtime, and more decoding steps give only slight gains in temporal consistency.
Limitations
- Only supports structure-preserving edits (style or appearance); cannot change object shape or add new objects in intermediate frames.
- Performance depends heavily on the image-editing model and structure detector; failures produce artifacts.
Experimental setup
Datasets: ShutterStock videos (100k) DAVIS
Benchmarks: DAVIS (40 object-centric videos) ShutterStock (30 unseen videos)
Metrics: prompt consistency (CLIP) temporal consistency (CLIP) warping error long-term temporal consistency PSNR SSIM LPIPS running time
Baselines: ControlNet per frame Tune-a-Video Text2Video-zero ControlVideo TokenFlow FILM RIFE
Builds on: MaskGiT ControlNet Tune-A-Video TokenFlow VQ-GAN
Closest papers by meaning
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024 · similarity 0.89)
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.88)
- Mask^2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long Video Generation (CVPR 2025 · similarity 0.88)
- EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Models (AAAI 2025 · similarity 0.88)
- Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation (ICLR 2025 · similarity 0.87)
- FreeMask: Rethinking the Importance of Attention Masks for Zero-Shot Video Editing (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.