Research map / Vision-language and multimodal / Cross-modal learning
Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PeskaVLP, a hierarchical surgical video-language pretraining framework with LLM-based text knowledge augmentation, clip-level visual self-supervision, and a DTW-based procedure-aware loss, pretrained on the SVL lecture-video dataset.
Problem
Surgical video-language pretraining suffers from a small dataset, noisy ASR narrations, limited phase-level text variability, and ignores the temporal procedural order of surgical actions.
Approach
Uses an LLM prompted with role behaviors (recipe, dictionary, summarizer) to augment narration, keystep, and abstract texts; adds SimSiam-style visual self-supervision at clip level; and applies a DTW-based hinge loss contrasting forward vs. temporally reversed child-text sequences at phase/video levels.
Key results
- Zero-shot phase recognition average accuracy 41.0 / F1 27.1 across four datasets, vs. 28.7 / 19.3 for HecVL.
- Zero-shot Cholec80 accuracy 45.1 vs. 41.7 for HecVL; StrasBypass70 46.7 vs. 26.9.
- 10-shot linear probing on Cholec80 reaches 61.9 accuracy vs. 56.1 for HecVL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add hierarchical LLM knowledge augmentation (ablation) | PeskaVLP pretraining · Zero-shot phase recognition across Cholec80, AutoLaparo, StrasBypass70, and BernBypass70 | accuracy: 41.7 -> 42.4; 23.3 -> 24.9; 26.9 -> 32.3; 22.8 -> 23.8 | Helped |
| Add clip-level visual self-supervision to language supervision (ablation) | PeskaVLP pretraining with knowledge augmentation · Zero-shot phase recognition across Cholec80, AutoLaparo, StrasBypass70, and BernBypass70 | accuracy: 42.4 -> 43.4; 24.9 -> 28.3; 32.3 -> 39.8; 23.8 -> 25.7 | Helped |
| Add phase/video procedure-aware LecNCE (ablation) | PeskaVLP with knowledge augmentation and clip visual self-supervision · Zero-shot phase recognition on AutoLaparo | accuracy: 28.3 -> 26.5 | Hurt |
| Add phase/video procedure-aware LecNCE (ablation) | PeskaVLP with knowledge augmentation and clip visual self-supervision · Zero-shot phase recognition on Cholec80, StrasBypass70, and BernBypass70 | accuracy: 43.4 -> 45.1; 39.8 -> 45.1; 25.7 -> 26.5 | Helped |
| Add phase/video procedure-aware LecNCE (ablation) | PeskaVLP with knowledge augmentation and clip visual self-supervision · 10-shot linear probing on AutoLaparo | accuracy: 53.8 -> 53.1 | Hurt |
| Add phase/video procedure-aware LecNCE (ablation) | PeskaVLP with knowledge augmentation and clip visual self-supervision · 10-shot linear probing on Cholec80, StrasBypass70, and BernBypass70 | accuracy: 60.4 -> 61.9; 63.5 -> 63.8; 62.2 -> 62.9 | Helped |
Surprising findings
- SurgVLP (contrastive on narrations only) underperforms MoCo visual self-supervision in linear probing, suggesting noisy narrations hurt visual representations.
- Adding the LecNCE procedure-aware objective degrades AutoLaparo zero-shot and linear-probing results, which the authors attribute to less routined procedures.
Limitations
- Clip-level visual self-supervision and DTW increase memory and computation cost.
- Knowledge augmentation for keystep and abstract texts depends on metadata and must be adapted for manually annotated datasets.
- Procedure-aware objective hurts performance on AutoLaparo.
Experimental setup
Datasets: SVL Cholec80 AutoLaparo MultiBypass140 SVL-Retrieval
Benchmarks: Cholec80 AutoLaparo StrasBypass70 BernBypass70 SVL-Retrieval
Metrics: accuracy F1-score Recall@1 Recall@5 Recall@10
Baselines: MIL-NCE CLIP SurgVLP HecVL Moco ImageNet pretraining
Builds on: CLIP InfoNCE HecVL SimSiam Dynamic Time Warping
Closest papers by meaning
- SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark (CVPR 2026 · similarity 0.87)
- Bridging Vision and Language for Robust Context-Aware Surgical Point Tracking: The VL-SurgPT Dataset and Benchmark (AAAI 2026 · similarity 0.86)
- MedHorizon: Towards Long-context Medical Video Understanding in the Wild (NeurIPS 2026 · similarity 0.86)
- MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments (CVPR 2025 · similarity 0.84)
- Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data (ICLR 2025 · similarity 0.84)
- PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology (AAAI 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.