Research map / Video / Video generation
EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EditBoard, a task-oriented benchmark with nine automatic metrics across fidelity, execution, consistency, and style dimensions (including three new fidelity metrics FF-α, FF-β, Semantic Score) to evaluate text-based video editing models on four task types.
Problem
Existing evaluations of text-based video editing models use few, inconsistently named metrics that output a single score, failing to reveal per-task strengths and weaknesses and lacking fidelity-focused metrics.
Approach
A task-oriented benchmark that organizes editing tasks into simple, intermediate, and difficult levels and reports nine metrics across four dimensions, adding three new fidelity metrics (FF-α and FF-β based on optical flow warping and flow angle, and a masked Semantic Score) that are validated against human preference.
Key results
- New fidelity metrics match human choices at 92.54% (FF-α), 89.93% (FF-β), and 95.24% (Semantic Score), with Pearson correlations of 0.9312, 0.9126, and 0.9417.
- Across five evaluated editing models, performance generally declines from simple (SOSA/SE) to difficult (SOMA/MOA) tasks; e.g., Ground-A-Video Success Rate drops from 0.8659 on SOSA to 0.8370 on MOA, and Video-P2P from 0.5156 to 0.3752.
- Reports a trade-off where models with higher execution (Control-A-Video) show worse fidelity than FateZero, which has better FF and Semantic Scores but lower Success Rate and CLIP Similarity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FateZero instead of Control-A-Video | video editing on SOSA · SOSA | Semantic Score | Helped |
| Use FateZero instead of Control-A-Video | video editing on SOSA · SOSA | Success Rate | No clear effect |
| Use FateZero's attention blending rather than Control-A-Video's latent manipulation | video editing models · All tasks | Fidelity | Helped |
| Use Control-A-Video's latent manipulation rather than FateZero's attention blending | video editing models · All tasks | Execution | Helped |
Surprising findings
- FateZero achieves better fidelity scores but lower execution, indicating a fidelity-execution trade-off among editing methods.
- Some models excel at complex tasks yet underperform on simpler tasks compared to baselines.
Limitations
- Evaluation covers only 10 videos per task and five models, a small test set.
- Warping-based FF-α fails for videos with significant object movement, requiring FF-β as a complement.
- Benchmark will only include more models as they are open-sourced.
Experimental setup
Datasets: DAVIS LOVEU-TGVE-2023 VBench
Benchmarks: EditBoard
Metrics: FF-α FF-β Semantic Score Success Rate CLIP Similarity Subject Consistency Background Consistency Aesthetic Quality Imaging Quality Pearson correlation
Baselines: FateZero Control-A-Video Ground-A-Video Video-P2P TokenFlow
Builds on: VBench CLIP FateZero LPIPS
Closest papers by meaning
- VE-Bench: Subjective-Aligned Benchmark Suite for Text-Driven Video Editing Quality Assessment (AAAI 2025 · similarity 0.91)
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024 · similarity 0.89)
- ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing (NeurIPS 2026 · similarity 0.88)
- I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing (NeurIPS 2024 · similarity 0.88)
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.88)
- MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.