Research map / Video / Video-language understanding
Efficient Motion-Aware Video MLLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EMA, a video MLLM that feeds compressed-domain H.264 GOPs (I-frame RGB plus motion vectors) through a motion-aware GOP encoder into Qwen2-7B, and introduces the MotionBench benchmark for motion understanding.
Problem
Video MLLMs typically use uniform frame sampling with image-level encoders, which is token-inefficient and weak at capturing motion.
Approach
Operate directly on compressed video streams: encode each GOP's I-frame RGB with a vision encoder and its sparse motion vectors with a small transformer, fuse them via cross-attention into one fixed-length token set per GOP, so token count matches a single frame.
Key results
- EMA reaches 75.8 accuracy on MSVD-QA, 58.5 on MSRVTT-QA and 52.1 on ActivityNet-QA, the best among listed baselines.
- On MotionBench, EMA averages 50.0% vs 36.0% for Video-LLaVA, using 648 visual tokens vs 2048 and about 3.08x lower inference time than Video-LLaVA.
- On VideoMME (w/o subs), the 32-GOP variant reaches 53.4 overall vs 52.6 for LongVA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add motion vector input with motion encoder (GOP encoder with motion) vs. RGB-only (ablation) | EMA · MotionBench | Avg. Acc.: 44.5 -> 49.2 | Helped |
| Warm up motion encoder on SSV2 motion vectors before stage 1 (ablation) | EMA · MotionBench | Avg. Acc.: 45.6 -> 49.2 | Helped |
| Cross-attention fusion instead of additive fusion of motion and frame features (ablation) | EMA · MotionBench | Avg. Acc.: 46.2 -> 49.2 | Helped |
| Cross-attention fusion instead of concatenation of motion and frame features (ablation) | EMA · MotionBench | Avg. Acc.: 47.0 -> 49.2 | Helped |
| Temporal mean pooling of motion features vs. full motion sequence as key/value (ablation) | EMA · MotionBench | Avg. Acc.: 49.0 (no agg.) vs 49.2 (pooled) | No clear effect |
| Learnable motion position embedding added before motion encoder (pre-fusion) vs. none (ablation) | EMA · MotionBench | Avg. Acc.: 44.8 -> 49.2 | Helped |
| Position embedding added before vs after motion encoder (ablation) | EMA · MotionBench | Avg. Acc.: 47.9 -> 49.2 | Helped |
| Motion encoder depth 1 layer vs 2 layers (2 to 4 layers gives similar results) (ablation) | EMA · MotionBench | Avg. Acc.: 47.2 (1 layer) vs 49.2 (2 layers) vs 48.8 (3 layers) | Mixed |
| GOP segments instead of decoded frame segments at same token count (ablation) | Qwen2-7B video MLLM · MotionBench | Avg. Acc.: 44.5 -> 49.2 (4 segments); 44.3 -> 50.0 (8 segments) | Helped |
| GOP segments instead of frame segments at same count on video QA (ablation) | Qwen2-7B video MLLM · MSRVTT-QA | Acc.: 52.6 -> 57.0 (4 segments); 56.7 -> 58.5 (8 segments) | Helped |
| EMA GOP encoder vs uniform frame-sampling baselines with more visual tokens | EMA · Inference time (MLLM generation) | Inference time per sample: 391.4ms (Video-LLaVA) vs 127.1ms (EMA) | Helped |
Surprising findings
- Motion-aware GOP input with 8 GOPs matches the token count of 8 frames while outperforming 8 denser decoded frames on MotionBench (50.0 vs 44.3).
Limitations
- MotionBench is small (2.3k videos) and manually constructed, with multiple-choice format only.
- Relies on re-encoding videos with H.264 at fixed keyframe interval and frame rate, limiting applicability to other codecs or raw streams.
- Long-video results are only reported on VideoMME and are not uniformly superior to all baselines.
Experimental setup
Datasets: LLaVA-558k Valley-702k Cauldron VideoChat2-IT Something-to-Something V2 MSVD-QA MSRVTT-QA ActivityNet-QA VideoMME
Benchmarks: MotionBench MSVD-QA MSRVTT-QA ActivityNet-QA VideoMME
Metrics: accuracy score inference time visual token count
Baselines: Video-LLaVA LLaMA-VID Chat-UniVi Video-LaVIT VideoChat2 LongVA MovieChat BT-Adapter
Builds on: LLaVA Video-LLaVA LLaMA-VID Video-LaVIT
Closest papers by meaning
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.87)
- MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation (NeurIPS 2025 · similarity 0.87)
- HIMMEL: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding (NeurIPS 2026 · similarity 0.86)
- FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding (NeurIPS 2025 · similarity 0.86)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.85)
- Towards Understanding Camera Motions in Any Video (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.