Research map / Video / Video generation
VIRES: Video Instance Repainting via Sketch and Text Guided Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VIRES, a DiT-based (Open-Sora) video instance repainting method guided by sketch sequences and text, with a Sequential ControlNet, standardized self-scaling, sketch attention, and a sketch-aware VAE decoder encoder, plus the VIRESET dataset.
Problem
Video instance editing with text and sketch guidance struggles with temporal consistency and precise alignment to the provided sketch sequence.
Approach
Builds sketch-conditioned control on a pretrained DiT text-to-video model: a video-aware Sequential ControlNet with standardized self-scaling of sketch features, sketch attention as a parallel branch in spatial self-attention, and a sketch-aware encoder injecting multi-level texture features into the VAE decoder.
Key results
- On VIRESET, VIRES reaches PSNR 23.87 and SSIM 77.87 versus 18.99 and 70.81 for the next best (RAVE), and leads on all five automatic metrics on both VIRESET and DAVIS.
- In user studies (25 AMT volunteers), VIRES gets 54.40% visual-quality preference and 43.20% text-alignment preference on VIRESET.
- Editing a 51-frame 512x512 clip takes about 323 seconds on a single A6000 GPU, roughly 1.6x slower than other diffusion methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIRES instead of RAVE | Video instance repainting · VIRESET | PSNR: 18.99 -> 23.87 | Helped |
| Use VIRES instead of RAVE | Video instance repainting · DAVIS | PSNR: 19.08 -> 25.36 | Helped |
| Add standardized self-scaling (ablation) | VIRES · VIRESET | PSNR: 22.72 -> 23.87 | Helped |
| Add self-scaling (ablation) | VIRES · VIRESET | PSNR: 23.09 -> 23.87 | Helped |
| Add sketch attention (ablation) | VIRES · VIRESET | PSNR: 22.89 -> 23.87 | Helped |
| Add sketch-aware encoder (ablation) | VIRES · VIRESET | PSNR: 22.94 -> 23.87 | Helped |
| Add standardized self-scaling (ablation) | VIRES · DAVIS | PSNR: 24.59 -> 25.36 | Helped |
| Add self-scaling (ablation) | VIRES · DAVIS | PSNR: 24.95 -> 25.36 | Helped |
| Add sketch attention (ablation) | VIRES · DAVIS | PSNR: 24.86 -> 25.36 | Helped |
| Add sketch-aware encoder (ablation) | VIRES · DAVIS | PSNR: 24.72 -> 25.36 | Helped |
Limitations
- High computational cost of DiT-based architecture; ~1.6x slower than other diffusion-based methods.
- Sparse sketch guidance and long-duration repainting require extra fine-tuning or iterative inference strategies.
Experimental setup
Datasets: VIRESET SA-V DAVIS
Benchmarks: VIRESET DAVIS
Metrics: PSNR SSIM Warp Error Frame Consistency Text Consistency user preference (VQE) user preference (TAE)
Baselines: VideoComposer Text2Video-Zero Rerender VidToMe RAVE
Builds on: Open-Sora ControlNet VideoComposer flow matching
Closest papers by meaning
- SketchVideo: Sketch-based Video Generation and Editing (CVPR 2025 · similarity 0.87)
- Keyframe-Guided Creative Video Inpainting (CVPR 2025 · similarity 0.87)
- Prune and Repaint: Content-Aware Image Retargeting for any Ratio (NeurIPS 2024 · similarity 0.86)
- Elevating Flow-Guided Video Inpainting with Reference Generation (AAAI 2025 · similarity 0.86)
- ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing (NeurIPS 2026 · similarity 0.85)
- CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.