Research map / Vision-language and multimodal / Multimodal LLMs

SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SurgCoT, a surgical video benchmark with a three-stage chain-of-thought protocol and five-tuple annotations (Question→Option→Knowledge→Clue→Answer), and evaluates 10 MLLMs under BL/KE/FC settings.

Problem

MLLMs' fine-grained spatiotemporal and causal reasoning on surgical videos is largely unexplored; existing surgical VQA benchmarks are frame/clip-level and lack progressive, chain-of-thought evaluation.

Approach

A three-stage progressive reasoning framework (video-level, clip-level, frame-level) with a five-tuple annotation protocol where Knowledge supplies clinical background and Clue supplies spatiotemporal evidence, and each stage's Answer conditions the next stage.

Key results

• GPT-5 reaches 87.58% average accuracy under the full-context (FC) setting, versus 80.54% at baseline (BL), while Claude-Sonnet-4.5 reaches 87.54% FC.
• Sub-question accuracy drops sharply for commercial models: GPT-5 scores 76.62% on main questions but 47.60% on Q3 in the KE setting.
• Qwen2.5-VL-7B gains about 8.23% from KE to FC, and Claude-Sonnet-4.5 improves about 13.44% with spatiotemporal clues.

Surprising findings

• Commercial models answer main questions correctly while failing intermediate sub-questions, indicating broken reasoning chains.
• Medical-specialized and open-source models lag commercial models, with MedGemma-27B-IT the strongest medical model.

Limitations

• Benchmark evaluated only in zero-shot settings with a single prompt template and no fine-tuning experiments.
• Annotation pipeline relies on automatic tools (YOLOv10, SAM2, ByteTrack) and ASR, with expert validation only on a subset.
• Data includes proprietary clinical archives that may not be publicly released.

Experimental setup

Datasets: ASVIDE YouTube surgical videos ten open-source surgical repositories proprietary clinical archives

Benchmarks: SurgCoT

Metrics: accuracy

Baselines: GPT-5 Gemini-2.5-Pro Claude-Sonnet-4.5 MedGemma-27B-IT Lingshu-7B LLaVA-Med-7B HuatuoGPT-Vision-7B InternVL-8B Qwen2.5-VL-7B Qwen3-VL-8B

Builds on: YOLOv10 SAM2 ByteTrack

Closest papers by meaning

• Bridging Vision and Language for Robust Context-Aware Surgical Point Tracking: The VL-SurgPT Dataset and Benchmark (AAAI 2026 · similarity 0.88)
• Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation (NeurIPS 2024 · similarity 0.87)
• MedHorizon: Towards Long-context Medical Video Understanding in the Wild (NeurIPS 2026 · similarity 0.87)
• M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding (ICLR 2026 · similarity 0.85)
• SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models (ICML 2026 · similarity 0.85)
• SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics (NeurIPS 2026 · similarity 0.85)