Research map / Vision-language and multimodal / Multimodal LLMs
VlogReward: Learning Multi-Dimensional Evaluation for Vlog Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines a six-dimension vlog evaluation taxonomy, builds a 100k-edit dataset and VRMBench, and trains VlogReward (Qwen2.5-VL-7B, SFT then GRPO) with an added inter-group comparison reward to fix GRPO's "direction blindness".
Problem
Automated evaluation of vlog editing plans is hard because quality criteria are subjective and multi-layered, and there are no standard criteria, datasets, benchmarks, or reward models for it.
Approach
Adds an inter-group comparison reward to GRPO, scoring whether each rollout's predicted total score preserves the ground-truth quality ordering across plans of the same raw footage, combined with a per-dimension score reward; this lets the model distinguish varied-quality edits and gives multi-dimensional scores plus feedback.
Key results
- VlogReward reaches 73.5% comparison accuracy and 67.6% best-of-N accuracy on VRMBench average, versus 69.4% and 65.0% for GRM-GRPO without the comparison reward.
- Gemini-3-Pro is the strongest closed baseline at 71.9% comparison accuracy; GPT-5 reaches 63.1%.
- On UltraFeedback, SR+CR raises GRPO's BoN accuracy from 51.2% to 53.6%.
Surprising findings
- Thinking-mode (chain-of-thought) output does not improve and is comparable or worse than direct answers for vlog reward modeling.
- Inter-group advantage alone (GRM-GRPO-Inter-Advantage) underperforms the score-reward baseline on comparison and BoN accuracy.
Limitations
- Limited to montage-style vlogs and a JSON plan representation without fine-grained stylistic controls.
- Ground-truth scores reflect expert consensus on inherently subjective criteria.
- Low frame rate and resolution may miss cinematographic detail.
Experimental setup
Datasets: VRM-Dataset-SFT VRM-Dataset-RL VRM-Dataset-OB UltraFeedback
Benchmarks: VRMBench UltraFeedback
Metrics: score accuracy comparison accuracy best-of-n accuracy
Baselines: GPT-4o GPT-5 o4-mini Gemini-3-Pro Qwen2.5-VL-72B Qwen3-VL-8B LLaVA-OneVision-7B LLaVA-NeXT-Video-7B InternVL3.5-14B Kimi-VL-A3B-Thinking-2506 MiMo-VL-7B-RL RRM-MSE
Builds on: GRPO Bradley-Terry reward model Qwen2.5-VL
Closest papers by meaning
- Vlogger: Make Your Dream A Vlog (CVPR 2024 · similarity 0.85)
- VE-Bench: Subjective-Aligned Benchmark Suite for Text-Driven Video Editing Quality Assessment (AAAI 2025 · similarity 0.85)
- EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Models (AAAI 2025 · similarity 0.84)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.84)
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities (ICML 2024 · similarity 0.84)
- VEU-Bench: Towards Comprehensive Understanding of Video Editing (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.