Research map / Video / Video-language understanding
Video-Oasis: Rethinking Evaluation of Video Understanding
Geuntaek Lim, Minho Shim, Sungjune Park, Jaeyun Lee, Inwoong Lee, Taeoh Kim, Dongyoon Wee, Yukyung Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AKS temporal grounding (16 frames) (ablation) | Eagle2.5 · Video-Oasis distilled set | accuracy: 31.5 -> 32.9 | Improved |
| add AKS temporal grounding (16 frames) (ablation) | Qwen3-VL-Instruct · Video-Oasis distilled set | accuracy: 27.8 -> 30.1 | Improved |
| add oracle ground-truth temporal grounding (ablation) | Eagle2.5 · Video-Oasis distilled samples | accuracy: 35.0 -> 50.8 | Improved |
| add oracle ground-truth temporal grounding (ablation) | Eagle2.5 · shortcut samples | accuracy: 78.0 -> 80.8 | No clear effect |
| use adaptive thinking (VideoAuto-R1) instead of always-on thinking (ablation) | Qwen3-VL-8B · Video-Oasis distilled set | accuracy: 34.6 -> 36.8 | Improved |
| switch from instruct to thinking mode (ablation) | Qwen3-VL-8B · Video-Oasis distilled set | accuracy: 33.8 -> 34.6 | Improved |
| oracle choice between instruct and thinking per question (ablation) | Qwen3-VL-8B · Video-Oasis distilled set | accuracy: 34.6 -> 46.2 | Improved |
| long-context SFT (Eagle2.5) over base | Qwen2.5-VL · Video-Oasis distilled set | accuracy: 29.2 -> 34.5 | Improved |
| RLVR with QA-only reward (Video-R1) | Qwen2.5-VL · Video-Oasis distilled set | accuracy: 29.2 -> 26.3 | Hurt |
| RLVR with QA plus grounding reward (VideoAuto-R1) | Qwen2.5-VL · Video-Oasis distilled set | accuracy: 29.2 -> 32.7 | Improved |
| add grounding reward in RLVR | Qwen2.5-VL · Global Narrative | accuracy: 21.2 -> 28.6 | Improved |
| filter shortcut samples with Video-Oasis | Qwen2.5-VL (7B) · 14 benchmarks | accuracy: 51.2 -> 29.2 | Hurt |
Closest papers by meaning
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.90)
- VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning? (ICLR 2026 · similarity 0.89)
- VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.89)
- Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (ICLR 2025 · similarity 0.89)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.89)
- Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
