Research map / Video / Video-language understanding
MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MotionBench, a 8,052-QA benchmark for fine-grained motion-level video understanding in VLMs, and a Through-Encoder Fusion method for video feature compression under limited LLM context length.
Problem
Existing video VLM benchmarks largely neglect fine-grained motion-level perception, and current VLMs can only process few frames, limiting the high frame rates needed for motion understanding.
Approach
Builds a motion-focused multiple-choice benchmark across six motion question types and multiple video sources, and proposes Through-Encoder Fusion, which groups adjacent frames and applies group-level self-attention throughout the visual encoder before spatial-temporal compression, rather than shallow fusion only at the compressor.
Key results
- Best existing model Qwen2VL-72B reaches 0.57 dev / 0.58 test average on MotionBench; most models score below 60%.
- TE Fusion on GLM-4V-9B with 16 frames and compression 4 reaches 0.58 dev average, state-of-the-art among listed models with a 9B LLM.
- At k=4 with 4 input frames, TE Fusion scores 51.0 on MotionBench and 72.1 on MVBench, improving over the uncompressed baseline (47.6 and 64.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Through-Encoder Fusion rather than 4-frame uncompressed baseline | GLM-4V-9B video model · MotionBench | score: 47.6 -> 51.0 | Helped |
| Use Through-Encoder Fusion rather than 4-frame uncompressed baseline | GLM-4V-9B video model · MVBench | score: 64.5 -> 72.1 | Helped |
| Use Through-Encoder Fusion rather than 4-frame uncompressed baseline | GLM-4V-9B video model · VideoMME short | score: 51.4 -> 61.0 | Helped |
| Use Through-Encoder Fusion rather than 4-frame uncompressed baseline | GLM-4V-9B video model · VideoMME medium | score: 41.0 -> 47.3 | Helped |
| Use Through-Encoder Fusion rather than 4-frame uncompressed baseline | GLM-4V-9B video model · VideoMME long | score: 38.3 -> 42.1 | Helped |
| Replace Through-Encoder Fusion with QFormer compression (ablation) | GLM-4V-9B video model · MotionBench | score: 51.0 -> 44.3 | Hurt |
| Replace Through-Encoder Fusion with Qwen2-VL pre-encoder fusion (ablation) | GLM-4V-9B video model · MVBench | score: 72.1 -> 65.6 | Hurt |
| Replace Through-Encoder Fusion with PLLaVA post-encoder fusion (ablation) | GLM-4V-9B video model · VideoMME short | score: 61.0 -> 58.9 | Hurt |
| Replace Through-Encoder Fusion with Kangaroo post-encoder fusion (ablation) | GLM-4V-9B video model · VideoMME medium | score: 47.3 -> 45.6 | Hurt |
| Replace Through-Encoder Fusion with PLLaVA post-encoder fusion (ablation) | GLM-4V-9B video model · VideoMME long | score: 42.1 -> 41.3 | Hurt |
| Use TE Fusion rather than Qwen2VL-72B | MotionBench dev evaluation · MotionBench dev | accuracy: 0.57 -> 0.58 | Helped |
| Use TE Fusion rather than Qwen2VL-72B | MotionBench test evaluation · MotionBench test | accuracy: 0.58 -> 0.58 | No clear effect |
Surprising findings
- Repetition Count is near random for most models, and models achieve high scores on Motion-related Objects, suggesting pretraining bias toward objects.
- Higher compression ratio (k=4) gave best results over k=2 in 3 of 4 datasets for TE Fusion.
Limitations
- Benchmarks only four-choice multiple choice with regex-based answer extraction.
- Test set labels withheld; evaluation requires an online submission platform.
- TE Fusion evaluated only with a straightforward group-attention variant on one backbone (GLM-4V-9B).
Experimental setup
Datasets: Panda-70M Pexels MedVid SportsSloMo HA-ViD Unity-rendered synthetic videos movie clips
Benchmarks: MotionBench MVBench LVBench VideoMME
Metrics: accuracy
Baselines: GPT-4o Gemini 1.5 Pro Qwen2VL-72B InternVL-40B PLLaVA-34B CogVLM2-Video GLM-4V-plus LLaVA-NeXT MiniCPM-V2.6 Oryx-34B QFormer Qwen2-VL
Builds on: Qwen2-VL PLLaVA Kangaroo QFormer GLM-4V
Closest papers by meaning
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.96)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.94)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.93)
- MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs (NeurIPS 2025 · similarity 0.91)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.91)
- PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding? (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.