Research map / Video / Video-language understanding
FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FAVOR-Bench, a benchmark of 1,776 videos with 8,184 multiple-choice QA pairs and open-ended caption evaluation (GPT-assisted and LLM-free) for fine-grained video motion understanding, plus FAVOR-Train, a 17K-video training set used to fine-tune Qwen2.5-VL-7B.
Problem
Existing video benchmarks evaluate motion at the event level and lack fine-grained motion evaluation, including ego-centric videos and open-ended description tasks; MLLMs struggle with detailed temporal motion dynamics.
Approach
A semi-automatic QA pipeline (DeepSeek-R1 generation, blind and single-frame filtering, manual verification) across six motion-centric tasks, paired with a novel LLM-free open-ended metric that extracts structured motion elements with NLTK and matches them with Sentence-BERT similarity and Kendall's Tau ordering.
Key results
- Gemini-1.5-Pro achieved the highest close-ended accuracy on FAVOR-Bench at 49.87%, while Qwen2.5-VL-72B reached 48.14% among open-source models.
- Fine-tuning Qwen2.5-VL-7B on FAVOR-Train improved close-ended accuracy by 1.37 points (40.76 to 42.13), LLM-free score from 48.46 to 56.33, and GPT correctness score by 0.27 (3.28 to 3.55).
- On MotionBench-Dev, Qwen2.5-VL-7B improved from 45.2 to 49.2 overall ALL accuracy after FAVOR-Train fine-tuning; TVBench average rose from 45.2 to 46.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-VL-7B with FAVOR-Train | Qwen2.5-VL-7B · FAVOR-Bench close-ended ALL | accuracy (%): 40.76 -> 42.13 | Helped |
| Fine-tune Qwen2.5-VL-7B with FAVOR-Train | Qwen2.5-VL-7B · FAVOR-Bench GPT-assisted correctness | score: 3.28 -> 3.55 | Helped |
| Fine-tune Qwen2.5-VL-7B with FAVOR-Train | Qwen2.5-VL-7B · FAVOR-Bench GPT-assisted detailedness | score: 3.41 -> 3.53 | Helped |
| Fine-tune Qwen2.5-VL-7B with FAVOR-Train | Qwen2.5-VL-7B · FAVOR-Bench LLM-Free | score (%): 48.46 -> 56.33 | Helped |
| Fine-tune Qwen2.5-VL-7B with FAVOR-Train | Qwen2.5-VL-7B · TVBench | AVG score: 46.2 -> 47.9 | Helped |
| Fine-tune Qwen2.5-VL-7B with FAVOR-Train | Qwen2.5-VL-7B · MotionBench-Dev | ALL accuracy: 33.6 -> 34.1 | Helped |
Surprising findings
- Claude-3.7-Sonnet scored highest on the GPT-assisted evaluation while Tarsier2-Recap-7B, a captioning-specialized model, scored highest on the LLM-free metric despite failing the close-ended task.
- Most models scored high on sequence ordering (over 85%) but below 40% on action matching, indicating recognition rather than ordering is the bottleneck.
Limitations
- Blind and single-frame filtering may inadvertently exclude some correctly-answerable questions by chance.
- Proprietary models were limited to 16 frames for videos longer than 16 seconds due to API restrictions.
- GPT-assisted evaluation relies on GPT-4o, which is costly and less reproducible.
Experimental setup
Datasets: Charades EgoTaskQA Koala-36M Charades-Ego EgoExo4D EgoExoLearn TV series and animation clips (self-collected) Kinetics ActivityNet MIT
Benchmarks: FAVOR-Bench TVBench MotionBench
Metrics: multiple-choice accuracy GPT-4o correctness score GPT-4o detailedness score LLM-free score (precision/recall/order)
Baselines: Gemini-1.5-Pro GPT-4o Claude-3.7-Sonnet LLaVA-Video-72B-Qwen2 InternVL2.5-78B Qwen2.5-VL-72B Tarsier2-Recap-7B VideoLLaMA3-7B
Builds on: MotionBench Koala-36M EgoTaskQA
Closest papers by meaning
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.91)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.90)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.90)
- Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract) (AAAI 2025 · similarity 0.89)
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.89)
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.