Research map / Video / Video generation
Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds Señorita-2M, a ~2M-pair instruction-based video editing dataset generated by four CogVideoX-based expert editors (global/local stylizer, inpainter, remover) applied to crawled Pexels videos, with CLIP-based filtering, and benchmarks end-to-end editing architectures trained on it.
Problem
End-to-end instruction-based video editing is limited by the lack of large, high-quality video editing pairs; existing datasets are small, local-only, or synthetic with low quality.
Approach
Use specialized video editing experts (built on CogVideoX with ControlNet-style conditioning, inpainting, and a negative-instruction remover) to synthesize edited counterparts of real Internet videos, filter failures with CLIP-feature classifiers and similarity thresholds, and LLM-rewrite prompts into instructions.
Key results
- The Señorita-based FF-Control-Edit model reaches Ewarp 9.42, CLIPScore 0.2895, and 53.17% user preference in the main comparison, versus InsV2V at Ewarp 16.50 and CLIPScore 0.1675.
- On object removal, the expert remover reaches Ewarp 4.21 and relevance 0.1554, beating ProPainter's 4.93 and 0.1685 on those metrics while ProPainter wins PSNR/SSIM.
- Ablation: training on Señorita-2M instead of InsV2V at 60K samples raises CLIPScore from 0.2366 to 0.2596.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Señorita-2M instead of InsV2V for the 60K training examples (ablation) | CogVideoX-5B editing model · DAVIS | CLIPScore: 0.2366 -> 0.2596 | Helped |
| use Señorita-2M instead of InsV2V for the 60K training examples (ablation) | CogVideoX-5B editing model · DAVIS | Temporal Consistency: 0.9712 -> 0.9783 | Helped |
| increase Señorita-2M training samples from 60K to 120K (ablation) | CogVideoX-5B editing model · DAVIS | Ewarp (10^-3): 8.44 -> 7.95 | Helped |
| increase Señorita-2M training samples from 60K to 120K (ablation) | CogVideoX-5B editing model · DAVIS | CLIPScore: 0.2596 -> 0.2641 | Helped |
| increase Señorita-2M training samples from 60K to 120K (ablation) | CogVideoX-5B editing model · DAVIS | Temporal Consistency: 0.9783 -> 0.9785 | Helped |
Surprising findings
- ProPainter achieves higher PSNR and SSIM than the proposed remover, because it leaves background pixels unchanged, although its removed regions are blurry.
Limitations
- Evaluation uses a single DAVIS subset with random prompts and CLIP-based metrics, which may not reflect true editing quality.
- Some expert baselines (e.g., VIVID-10M, COCOCO) were not open-sourced, limiting direct comparison.
Experimental setup
Datasets: WebVid-10M Pexels videos DAVIS InsV2V Omni-Edit
Benchmarks: DAVIS
Metrics: Ewarp CLIPScore temporal consistency user preference PSNR SSIM LPIPS MSE
Baselines: TokenFlow Flatten AnyV2V InsV2V ProPainter VIVID-10M AVID CoCoCo
Builds on: CogVideoX ControlNet InstructPix2Pix Grounded-SAM2 WebVid-10M
Closest papers by meaning
- Transforming Image Editors into Video Editors (NeurIPS 2026 · similarity 0.88)
- VE-Bench: Subjective-Aligned Benchmark Suite for Text-Driven Video Editing Quality Assessment (AAAI 2025 · similarity 0.88)
- VEU-Bench: Towards Comprehensive Understanding of Video Editing (CVPR 2025 · similarity 0.88)
- VicEdit: Learning to Edit Videos from Visual In-Context Examples (NeurIPS 2026 · similarity 0.88)
- Learning Action and Reasoning-Centric Image Editing from Videos and Simulation (NeurIPS 2024 · similarity 0.87)
- EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.