Research map / Vision-language and multimodal / Multimodal LLMs

Bridging Vision and Language for Robust Context-Aware Surgical Point Tracking: The VL-SurgPT Dataset and Benchmark

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces VL-SurgPT, a multimodal in vivo surgical point tracking dataset with coordinates and point-status text labels, plus TG-SurgPT, a text-guided extension of Track-On that improves tracking under adverse visual conditions.

Problem

Surgical point tracking fails under smoke, specular reflection, occlusion and tissue deformation, yet existing surgical tracking datasets provide only coordinates without semantic descriptions of point status.

Approach

Pair in vivo ICG-marked point trajectories with per-point textual status labels (e.g., Smoke Obscuration, Reflection) and use a frozen CLIP text encoder with cross-modal attention on top of Track-On so that predicted status text guides coordinate refinement at inference.

Key results

• TG-SurgPT reaches 62.88 AJ and 11.02 px EPE on the tissue subset, versus 58.55 AJ and 13.79 px EPE for Track-On.
• On the instrument subset TG-SurgPT achieves 49.52 AJ and 89.79 OA, versus 46.97 AJ and 85.07 OA for Track-On.
• TG-SurgPT runs at 9.72 fps, below Track-On's 10.85 fps and BootsTAP's 14.25 fps.

Surprising findings

• Cauterization smoke scenario yields the highest mean tracking accuracy (58.62%) despite expectations of visual impairment, per the paper's scenario analysis.
• Instrument Out of View status is the poorest-predicted class (48.1%).

Limitations

• Dataset limited to gastrointestinal procedures on the da Vinci system.
• TG-SurgPT at 9.72 fps does not meet real-time clinical requirements.
• Status prediction accuracy is low for some categories such as Out of View.

Experimental setup

Datasets: VL-SurgPT

Benchmarks: VL-SurgPT

Metrics: AJ average position accuracy (<delta_avg^x) occlusion accuracy end point error fps

Baselines: RAFT SEA-RAFT TAPIR BootsTAP CoTracker3 MFT MFTIQ Track-On

Builds on: Track-On CLIP MFT STIR TAP-Vid

Closest papers by meaning

• CholecTrack20: A Multi-Perspective Tracking Dataset for Surgical Tools (CVPR 2025 · similarity 0.89)
• SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark (CVPR 2026 · similarity 0.88)
• SurgVista: Long-Horizon Surgical World Modeling with Plausible Instrument-Tissue Dynamics (NeurIPS 2026 · similarity 0.86)
• Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation (NeurIPS 2024 · similarity 0.86)
• MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments (CVPR 2025 · similarity 0.85)
• U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding (ICLR 2026 · similarity 0.83)