Research map / Detection and segmentation / Weakly supervised detection and segmentation
Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RASO, an open-set surgical object recognition foundation model for images and videos, pretrained on weakly-supervised tag-image-text triplets automatically generated from surgical lecture videos, with a temporal-attention fusion layer for video.
Problem
Surgical object recognition is closed-set and limited by the scarcity of large annotated surgical datasets, and existing surgical segmentation and VLM models have weak open-set recognition and rely on manual prompts.
Approach
Builds a weakly-supervised data engine and label engine that turn unannotated WebSurg lecture videos (speech transcripts via WhisperX, NER/semantic parsing, GPT-4o annotations) into tag-image-text triplets, and adapts RAM with a temporal-attention fusion layer for video recognition.
Key results
- Zero-shot image recognition: RASO improves mAP over the best baseline by 2.9 (CholecT50), 4.5 (Cholec80), 10.6 (RSS) and 7.2 (GraSP) points per the abstract; RASO reaches 22.6 mAP on CholecT50 and 65.3 mAP on RSS.
- Supervised CholecT50 triplet recognition: RASO reaches 57.5 mAP overall, surpassing Rendezvous (56.4).
- Video temporal fusion achieves 61.3 mAP at 100.8 ms latency versus 58.8 mAP at 398.8 ms for the image-based pipeline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with GPT-4o-generated annotations (ablation) | RASO · CholecT50 | zero-shot image mAP: 20.6 -> 22.6 | Helped |
| fine-tune with GPT-4o-generated annotations (ablation) | RASO · Cholec80 | zero-shot image mAP: 25.0 -> 37.8 | Helped |
| fine-tune with GPT-4o-generated annotations (ablation) | RASO · RSS (Endovis18) | zero-shot image mAP: 53.3 -> 58.8 | Helped |
| fine-tune with GPT-4o-generated annotations (ablation) | RASO · GraSP | zero-shot image mAP: 16.4 -> 20.0 | Helped |
| add pretraining (ablation) | RASO · CholecT50, supervised image | All triplets mAP: 54.8 -> 57.5 | Helped |
| add temporal fusion instead of average pooling (ablation) | RASO video · CholecT50, supervised video | Verb mAP: 68.5 -> 74.5 | Helped |
| add temporal fusion instead of average pooling (ablation) | RASO video · CholecT50, supervised video | All triplets mAP: 60.1 -> 61.3 | Helped |
| add pretraining (ablation) | RASO video with temporal fusion · CholecT50, supervised video | All triplets mAP: 57.4 -> 61.3 | Helped |
| use temporal fusion for video rather than framewise image inference and tag union (ablation) | RASO · CholecT50 video | mAP: 58.8% -> 61.3% | Helped |
| use temporal fusion for video rather than framewise image inference and tag union (ablation) | RASO · CholecT50 video | latency: 398.8 ms -> 100.8 ms | Helped |
Surprising findings
- Clinician agreement with GPT-4o tag annotations is low, ranging from 0.19 to 0.35 across 20 frequent tags.
Limitations
- GPT-4o annotations show low agreement with clinician annotations, so the VLM-generated labels are noisy.
- Zero-shot results on Cholec80 are marginally below SurgVLP for the w/o FT variant.
- Segmentation-focused datasets were converted to tag annotations from mask presence, which may be imperfect.
Experimental setup
Datasets: WebSurg lecture videos (2,200 procedures) CholecT50 Cholec80 GraSP EndoVis18 RSS
Benchmarks: CholecT50 Cholec80 EndoVis18 RSS GraSP
Metrics: mAP precision recall F-beta (beta=0.5)
Baselines: CLIP BiomedCLIP PubMedCLIP SurgVLP Tripnet Attention Tripnet Rendezvous
Builds on: Recognize Anything Model (RAM) Tag2Text CLIP Swin Transformer WhisperX
Closest papers by meaning
- SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation (AAAI 2024 · similarity 0.86)
- Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation (NeurIPS 2024 · similarity 0.84)
- CholecTrack20: A Multi-Perspective Tracking Dataset for Surgical Tools (CVPR 2025 · similarity 0.84)
- SurgCoT: Advancing Spatiotemporal Reasoning in Surgical Videos through a Chain-of-Thought Benchmark (CVPR 2026 · similarity 0.82)
- EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding (NeurIPS 2025 · similarity 0.82)
- MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.