Research map / Detection and segmentation / Open-set and few-shot detection
SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SurgicalSAM, an end-to-end efficient-tuning of SAM for class-promptable surgical instrument segmentation, using prototype-based class prompts and contrastive prototype learning on EndoVis2017/2018.
Problem
Zero-shot SAM generalises poorly to surgical instruments and needs precise point/box prompts from specialist detectors, creating complex multi-stage pipelines for surgical instrument segmentation.
Approach
Replace explicit point/box prompts with a lightweight prototype-based class prompt encoder that turns learnable class prototypes into dense and sparse prompt embeddings for SAM's mask decoder, with a prototype contrastive loss (infoNCE-style) to increase inter-class discrimination; only the prompt encoder and mask decoder are tuned while the ViT-H image encoder stays frozen.
Key results
- On EndoVis2018, SurgicalSAM reaches 80.33 Challenge IoU and 58.87 mc IoU with 4.65M tunable parameters, above MATIS Frame (82.37 is higher on Challenge IoU, so it is SOTA only on mc IoU vs. specialists) and all zero-shot SAM baselines.
- On EndoVis2017, SurgicalSAM achieves 69.94 Challenge IoU and 67.03 mc IoU, surpassing MATIS Frame's 68.79/37.30 on mc IoU by a large margin.
- Training uses less than 1/6 of MATIS Frame's GPU memory at the same batch size and is over 10x faster in training speed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| tune SAM with SurgicalSAM's prototype-based class prompt encoder and mask decoder | SAM ViT-H · EndoVis2018 | mc IoU: 48.65 -> 58.87 | Helped |
| tune SAM with SurgicalSAM's prototype-based class prompt encoder and mask decoder | SAM ViT-H · EndoVis2017 | mc IoU: 55.26 -> 67.03 | Helped |
| add contrastive prototype learning rather than using fixed class prototypes (ablation) | SurgicalSAM · EndoVis2018, n=2 | Challenge IoU: 76.38 -> 80.33 | Helped |
| add contrastive prototype learning rather than using fixed class prototypes (ablation) | SurgicalSAM · EndoVis2018, n=2 | mc IoU: 53.95 -> 58.87 | Helped |
| vary the number of tokens per class from 2 to 4, 6, or 8 (ablation) | SurgicalSAM · EndoVis2018 | Challenge IoU: 80.33; 79.46; 79.67; 80.10 | No clear effect |
| vary the number of tokens per class from 2 to 4, 6, or 8 (ablation) | SurgicalSAM · EndoVis2018 | mc IoU: 58.87; 58.40; 56.97; 58.30 | No clear effect |
| add dense prompt embeddings (ablation) | SurgicalSAM · EndoVis2018 | Challenge IoU: 23.61 -> 80.33 | Helped |
| add dense prompt embeddings (ablation) | SurgicalSAM · EndoVis2018 | mc IoU: 14.13 -> 58.87 | Helped |
| add sparse prompt embeddings (ablation) | SurgicalSAM · EndoVis2018 | Challenge IoU: 78.58 -> 80.33 | Helped |
| add sparse prompt embeddings (ablation) | SurgicalSAM · EndoVis2018 | mc IoU: 58.24 -> 58.87 | Helped |
| add positive and negative embeddings (ablation) | SurgicalSAM · EndoVis2018 | Challenge IoU: 77.69 -> 80.33 | Helped |
| add positive and negative embeddings (ablation) | SurgicalSAM · EndoVis2018 | mc IoU: 57.47 -> 58.87 | Helped |
Surprising findings
- Removing dense prompt embeddings drops Challenge IoU from 80.33 to 23.61 on EndoVis2018, far more than removing sparse embeddings.
- Oracle GT bounding-box prompts to SAM (88.04 Challenge IoU) exceed SurgicalSAM, showing headroom if precise boxes were available.
Limitations
- Inference with online feature extraction is only about 1.7 fps, limited by the frozen ViT-H encoder.
- Evaluated only on two surgical datasets with seven or nine instrument categories, and class prompts require known class IDs.
Experimental setup
Datasets: EndoVis2018 EndoVis2017
Benchmarks: EndoVis2018 EndoVis2017
Metrics: Challenge IoU IoU mc IoU per-class IoU training speed GPU memory inference speed
Baselines: TernausNet MF-TAPNet Dual-MF ISINet TraSeTr S3Net MATIS Frame MaskTrack-RCNN + SAM Mask2Former + SAM Track Anything PerSAM PerSAM (Fine-Tune)
Builds on: Segment Anything Model infoNCE TernausNet Track Anything PerSAM
Closest papers by meaning
- Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data (ICLR 2025 · similarity 0.86)
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.84)
- Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation (CVPR 2025 · similarity 0.83)
- SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction (AAAI 2024 · similarity 0.83)
- BLO-SAM: Bi-level Optimization Based Finetuning of the Segment Anything Model for Overfitting-Preventing Semantic Segmentation (ICML 2024 · similarity 0.83)
- Unleashing the Potential of SAM for Medical Adaptation via Hierarchical Decoding (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.