Research map / Video / Video generation
Generative Video Propagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenProp, a generative video propagation framework that propagates first-frame edits through an image-to-video diffusion model guided by a selective content encoder, trained on synthetic data from video instance segmentation datasets.
Problem
Video editing and related tasks (object removal, insertion, replacement, tracking) typically require dense per-frame masks or task-specific retraining, and propagation methods based on optical flow or depth suffer from error accumulation and limited generality.
Approach
Encode the original video with a selective content encoder (a ControlNet-like copy of the I2V model's initial blocks) and propagate first-frame edits with a frozen I2V generator; a region-aware loss with gradient penalty on the edited region plus a mask prediction decoder disentangle edited from preserved content, and synthetic data from instance segmentation datasets covers multiple tasks.
Key results
- On the Challenging Test Set, GenProp achieves PSNRm 32.163, CLIP-T 0.3336, CLIP-I 0.9904, outperforming AnyV2V, Pika, ReVideo and InsV2V.
- In an MTurk study with 121 participants, users preferred GenProp over ReVideo for alignment 84.44% and quality 82.22% on the Challenging Test Set (ReVideo row of Table 1).
- For object removal, GenProp reaches CLIP-I 0.9879 versus 0.9809 for SAM+ProPainter and 0.9728 for ReVideo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GenProp instead of the strongest listed baseline, Pika | video editing · Classic Test Set | PSNRm: 32.568 -> 33.837 | Helped |
| use GenProp instead of Pika | video editing · Classic Test Set | CLIP-I: 0.9923 -> 0.9825 | Hurt |
| use GenProp instead of Pika | video editing · Challenging Test Set | CLIP-T: 0.3023 -> 0.3336 | Helped |
| use GenProp instead of SAM + Propainter | object removal · Object removal | CLIP-I: 0.9809 -> 0.9879 | Helped |
| add the Mask Prediction Decoder (MPD) (ablation) | GenProp · Challenging Test Set | CLIP-T: 0.3252 -> 0.3316 | Helped |
| add the Mask Prediction Decoder (MPD) (ablation) | GenProp · Challenging Test Set | CLIP-I: 0.9834 -> 0.9872 | Helped |
| add the Region-Aware Loss (RA Loss) (ablation) | GenProp · Challenging Test Set | CLIP-T: 0.3261 -> 0.3316 | Helped |
| add the Region-Aware Loss (RA Loss) (ablation) | GenProp · Challenging Test Set | CLIP-I: 0.9825 -> 0.9872 | Helped |
| add Color Fill augmentation (ablation) | GenProp · object tracking | tracking accuracy | Helped |
Surprising findings
- Pika scores better frame consistency on the Classic Test Set, since its bounding box works well when object shape is unchanged.
- SAM 2 produces more precise tracking masks than GenProp, though GenProp tracks object effects like reflections and shadows more consistently.
Limitations
- Slower than real-time tracking methods like SAM 2 and less precise masks for tracking.
- Currently supports only first-frame edits; extending to multiple keyframe edits is left for future work.
- Evaluation relies on small custom test sets (30 and 15 videos) and a user study with commercial baselines.
Experimental setup
Datasets: YouTube-VOS SAV (SAM 2 dataset) internal video instance segmentation dataset DAVIS (via TGVE) Pexels videos Adobe Stock videos
Benchmarks: Classic Test Set (TGVE DAVIS subset) Challenging Test Set (30 Pexels/Adobe Stock videos) Object removal set (15 complex videos)
Metrics: PSNRm CLIP-T CLIP-I user preference %
Baselines: InsV2V AnyV2V Pika ReVideo SAM 2 + ProPainter
Builds on: ControlNet Sora-style DiT video generation Stable Video Diffusion SAM 2
Closest papers by meaning
- Generative Omnimatte: Learning to Decompose Video into Layers (CVPR 2025 · similarity 0.86)
- GenFusion: Closing the Loop between Reconstruction and Generation via Videos (CVPR 2025 · similarity 0.86)
- Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation (CVPR 2025 · similarity 0.86)
- GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control (CVPR 2025 · similarity 0.86)
- LongDiff: Training-Free Long Video Generation in One Go (CVPR 2025 · similarity 0.86)
- ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.