Research map / Vision-language and multimodal / Multimodal LLMs

SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SciVideoBench, a 1,000-question multiple-choice benchmark built from JoVE experimental videos across physics, chemistry, biology, and medicine, and evaluates 21+ LMMs, finding Gemini-2.5-Pro leads at 64.3% while open-source models trail.

Problem

Existing video benchmarks are saturated or target general or college-level content where perception dominates and visual evidence is often not required, so they do not measure research-level scientific video reasoning.

Approach

A research-level scientific video benchmark where questions are generated by a multi-agent Gemini 2.5 Pro pipeline from videos, transcripts, and papers, then verified by humans, with questions split into conceptual, hypothetical, and quantitative reasoning types and checked via blind (vision-removed) baselines.

Key results

• Gemini-2.5-Pro achieves 64.30% overall accuracy on SciVideoBench, while the best open-source model InternVL-3-78B-Instruct reaches 38.80%.
• Human graduate students score 17.40% in a closed-book setting, and blind GPT-4o scores 15.80%.
• Chain-of-thought prompting raises Gemini-1.5-Pro overall accuracy by +21.10 points (27.50% to 48.60%).

Surprising findings

• Open-source models often score lower with chain-of-thought overall, while quantitative accuracy improves.
• Blind (video-removed) models score near chance, yet the benchmark is claimed to require visual information.

Limitations

• Human evaluation covers only three graduate students per discipline with a closed-book setting.
• Questions are generated by Gemini 2.5 Pro, which is also evaluated on the benchmark, creating a possible bias.
• Only 241 videos from one source (JoVE) underpin the 1,000 questions.

Experimental setup

Datasets: JoVE videos JoVE manuscripts

Benchmarks: SciVideoBench

Metrics: accuracy

Baselines: GPT-4o Gemini-1.5-Pro Gemini-2.0-Flash Gemini-2.5-Flash Qwen2.5-VL InternVL-3 InternVideo2.5-Chat-8B LLaVA-OneVision LongVA Qwen2.5-Omni-7B

Builds on: LMM-Eval

Closest papers by meaning

• TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models (ICLR 2025 · similarity 0.92)
• VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning? (ICLR 2026 · similarity 0.91)
• VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding (NeurIPS 2025 · similarity 0.90)
• MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.90)
• Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition (ICML 2024 · similarity 0.90)
• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.90)