Research map / Video / Video-language understanding

MedHorizon: Towards Long-context Medical Video Understanding in the Wild

NeurIPS 2026 · Video › Video-language understanding · approach: Video LLMs with fine-grained understanding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MedHorizon, a benchmark of 1,253 evidence-grounded multiple-choice questions over 759 hours of full-length medical procedure videos, showing that the best MLLM reaches only 41.1% accuracy.

Problem

Long-context medical video understanding requires finding sparse, subtle evidence in full-length clinical procedures before reasoning over it, but existing medical video benchmarks assume the evidence is already localized in images, short clips, or pre-segmented videos.

Approach

A benchmark that keeps full-length clinical procedure videos intact and builds evidence-anchored multiple-choice questions with strengthened distractors, human verification, and measured evidence sparsity (0.166% evidence frames on average), then diagnoses failure modes such as non-monotonic frame scaling and attention drift.

Key results

• The best model, Gemini-3.1-Pro-Preview, reaches 41.1% overall accuracy on MedHorizon, while most open-source and medical MLLMs remain around the mid-20s.
• Evidence is extremely sparse: 0.166% mean evidence ratio (1.7 evidence frames per 1,000 frames), about 169x sparser than QVHighlights.
• Text-only accuracy mirrors the 25% random baseline; MedGRPO improves from 13.6% to 33.9% on Semantic tasks when given oracle GT evidence images instead of uniform frames.

Surprising findings

• Performance does not scale reliably with more sampled frames; MedGRPO accuracy drops as frames increase from 4 to 512.
• Attention in medical video models is diffuse rather than concentrated on a few salient frames, with Qwen3-VL and Lingshu needing 76.7% and 85.6% of temporal units to reach 90% attention mass.

Limitations

• Limited by public-video diversity and annotation quality.
• Multiple-choice protocol cannot fully capture free-form clinical reporting.
• Small per-task question counts (e.g., 26 for WR, RSR, CPR) make some task accuracies unstable.

Experimental setup

Datasets: AutoLaparo Galar GraSP MultiBypass PitVQA REAL-Colon TMVP US-Study

Benchmarks: MedHorizon

Metrics: accuracy

Baselines: GPT-5.4 GPT-5.4-mini Gemini-3.1-Pro-Preview Gemini-2.5-Flash Kimi-K2.5 Qwen3.5-Plus Qwen3.6-Plus Qwen3.5-Flash Qwen3-VL-4B Qwen3-VL-8B InternVL3.5-8B LLaVA-Video-72B

Builds on: ViLAMP MedGRPO AKS

Closest papers by meaning

• Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.89)
• Medical thinking with multiple images (ICLR 2026 · similarity 0.89)
• M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding (ICLR 2026 · similarity 0.88)
• OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.88)
• MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.88)
• U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding (ICLR 2026 · similarity 0.87)