Research map / Video / Video generation
VE-Bench: Subjective-Aligned Benchmark Suite for Text-Driven Video Editing Quality Assessment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VE-Bench DB, a MOS-annotated video quality database of 1,170 edited videos from 8 text-driven editing models, and VE-Bench QA, a DOVER-based network with BLIP text-video alignment and source-target relation branches that correlates better with human ratings than prior metrics.
Problem
Existing metrics for text-driven video editing (CLIP scores, warp errors, FVD, etc.) and natural-video VQA methods do not align well with human perception and do not jointly model text-video alignment and source-edited video relationships.
Approach
Builds a subjective-aligned VQA dataset for edited videos with MOS labels and a metric network that combines DOVER-style aesthetic/technical branches, a BLIP-based text-video alignment branch with temporal adapter, and a source-target relevance branch fused by concatenation.
Key results
- On VE-Bench DB 10-fold, VE-Bench QA reaches SROCC 0.7415 and PLCC 0.7330, versus 0.6889 SROCC for StableVQA and 0.6119 for DOVER.
- On T2VQA-DB, VE-Bench QA reaches SROCC 0.8179 and PLCC 0.8227, above T2VQA (0.7965 SROCC).
- Ablation shows BLIP text branch raises SROCC from 0.6119 (DOVER) to 0.7171, and concatenation fusion gives the best result (0.7415).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VE-Bench QA instead of StableVQA | video quality assessment on VE-Bench DB · VE-Bench DB 10-fold | SROCC: 0.6889 -> 0.7415 | Helped |
| Use VE-Bench QA instead of T2VQA | video quality assessment on T2VQA-DB · T2VQA-DB | SROCC: 0.7965 -> 0.8179 | Helped |
| Use BLIP instead of CLIP for video-text alignment (ablation) | VE-Bench QA · VE-Bench DB 10-fold | SROCC: 0.6379 -> 0.7171 | Helped |
| Add a temporal feature extractor (VSwin) (ablation) | VE-Bench QA with BLIP text branch · VE-Bench DB 10-fold | SROCC: 0.7171 -> 0.7187 | Helped |
| Use MVD instead of no temporal feature extractor (ablation) | VE-Bench QA with BLIP text branch · VE-Bench DB 10-fold | SROCC: 0.7171 -> 0.7228 | Helped |
| Use Uformer instead of no temporal feature extractor (ablation) | VE-Bench QA with BLIP text branch · VE-Bench DB 10-fold | SROCC: 0.7171 -> 0.7317 | Helped |
| Add source-target feature fusion using MCA (ablation) | VE-Bench QA with Uformer temporal extractor · VE-Bench DB 10-fold | SROCC: 0.7317 -> 0.7330 | Helped |
| Concatenate source and edited-video features instead of using no fusion (ablation) | VE-Bench QA with Uformer temporal extractor · VE-Bench DB 10-fold | SROCC: 0.7317 -> 0.7415 | Helped |
| Use concatenation instead of MCA for feature fusion (ablation) | VE-Bench QA with Uformer temporal extractor · VE-Bench DB 10-fold | SROCC: 0.7330 -> 0.7415 | Helped |
| Add the additional parameters (ablation) | VE-Bench QA · VE-Bench DB 10-fold | SROCC: 0.7251 -> 0.7415 | Helped |
Surprising findings
- Zero-shot objective metrics such as S_edit and CLIP-F correlate weakly with human MOS (SROCC around 0.17-0.23).
- Models struggle with removal edits and shape/size edits, with FateZero as an exception attributed to its shape-aware attention blending.
Limitations
- Dataset is modest in size (169 source videos, 32-frame clips, 8 editing methods).
- Source-target branch cannot be applied directly to datasets lacking source videos, requiring removal for T2VQA-DB testing.
Experimental setup
Datasets: DAVIS Kinetics-700 Sintel Spring VE-Bench DB T2VQA-DB
Benchmarks: VE-Bench DB T2VQA-DB
Metrics: SROCC PLCC KRCC RMSE
Baselines: CLIP-F S_edit PickScore FastVQA StableVQA DOVER CLIPSim ViCLIP UMTScore T2VQA
Builds on: DOVER BLIP Uniformer PLCC loss rank loss
Closest papers by meaning
- EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Models (AAAI 2025 · similarity 0.91)
- VEU-Bench: Towards Comprehensive Understanding of Video Editing (CVPR 2025 · similarity 0.91)
- ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing (NeurIPS 2026 · similarity 0.89)
- Q-Bench-Video: Benchmark the Video Quality Understanding of LMMs (CVPR 2025 · similarity 0.88)
- Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists (NeurIPS 2025 · similarity 0.88)
- Video-Bench: Human-Aligned Video Generation Benchmark (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.