Research map / Video / Video segmentation and tracking
Multi-view Masked Contrastive Representation Learning for Endoscopic Video Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M2CRL, a self-supervised endoscopic video pretraining framework that combines multi-view tube masking (frame-aggregated attention-guided global views and random tube masking on local views) with self-distillation contrastive learning and pixel reconstruction.
Problem
Contrastive self-supervised pretraining for endoscopic videos yields discriminative but fine-grained-poor representations, which hurts pixel-level tasks such as segmentation and detection; generic video masking ignores endoscopy's inter-frame instability and low lesion contrast.
Approach
Combine self-distillation contrastive learning (Endo-FM style global/local views) with masked video reconstruction, where global views use a frame-aggregated attention-guided tube mask derived from the teacher's last-block class-token attention and local views use a random tube mask at 90% ratio.
Key results
- On PolypDiag classification, M2CRL reaches 94.2% F1 vs 90.7% for Endo-FM and 89.6% for VideoMAE V2.
- On CVC-12k segmentation, M2CRL reaches 81.4% Dice vs 73.9% for Endo-FM (+7.5) and 81.0% for VideoMAE V2.
- On KUMC detection, M2CRL reaches 86.3% F1 vs 84.1% for Endo-FM and 84.2% for VideoMAE V2; 88.7 F1 on Cholec80 phase recognition vs 87.5 for Endo-FM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine multi-view masked modeling with contrastive learning (ablation) | M2CRL · PolypDiag | F1: 90.7 ± 0.4 -> 94.2 ± 0.7 | Helped |
| Combine multi-view masked modeling with contrastive learning (ablation) | M2CRL · CVC-12k | Dice: 73.9 ± 1.2 -> 81.4 ± 0.8 | Helped |
| Combine multi-view masked modeling with contrastive learning (ablation) | M2CRL · KUMC | F1: 84.1 ± 1.3 -> 86.3 ± 0.8 | Helped |
| Use FAGTM on global views and random tube masking on local views (ablation) | M2CRL · PolypDiag | F1: 91.3 ± 0.3 -> 94.2 ± 0.7 | Helped |
| Use FAGTM on global views and random tube masking on local views (ablation) | M2CRL · CVC-12k | Dice: 77.7 ± 0.4 -> 81.4 ± 0.8 | Helped |
| Use FAGTM on global views and random tube masking on local views (ablation) | M2CRL · KUMC | F1: 84.9 ± 1.0 -> 86.3 ± 0.8 | Helped |
| Set the FAGTM attention threshold γ to 0.6 (ablation) | M2CRL · PolypDiag | F1: 91.6 ± 0.5 -> 94.2 ± 0.7 | Helped |
| Set the FAGTM attention threshold γ to 0.6 (ablation) | M2CRL · CVC-12k | Dice: 80.7 ± 0.7 -> 81.4 ± 0.8 | Helped |
| Use the teacher's last block for FAGTM (ablation) | M2CRL · CVC-12k | Dice: 76.6 ± 1.5 -> 81.4 ± 0.8 | Helped |
| Set global and local masking ratios to 90% rather than 75% (ablation) | M2CRL · CVC-12k | Dice: 74.6 ± 1.4 -> 81.4 ± 0.8 | Helped |
| Use RGB pixel regression instead of feature distillation as the reconstruction target (ablation) | M2CRL · CVC-12k | Dice: 77.9 ± 1.6 -> 81.4 ± 0.8 | Helped |
| Use ℓ2 rather than ℓ1 reconstruction loss (ablation) | M2CRL · CVC-12k | Dice: 81.4 ± 0.8 -> 82.0 ± 0.7 | No clear effect |
Surprising findings
- Masked modeling alone improves segmentation (73.9 to 80.5 Dice) but substantially lowers classification (90.7 to 85.7 F1), showing a trade-off between discriminative and pixel-level performance.
- Random masking at 75% ratio performs worse than 90% for video, and 95% ratio gives comparable but lower results.
Limitations
- Uses only RGB video streams; no audio or text modalities.
- Extensive pretraining requires significant energy and high-performance GPU resources.
- Evaluated on only three downstream tasks with small public datasets (e.g., CVC-12k with 29 videos).
Experimental setup
Datasets: Colonoscopic SUN-SEG LDPolypVideo Hyper-Kvasir Kvasir-Capsule CholecTriplet Renji-Hospital PolypDiag CVC-12k KUMC
Benchmarks: PolypDiag CVC-12k KUMC Cholec80
Metrics: F1 Dice F1 (detection)
Baselines: Endo-FM VideoMAE VideoMAE V2 DropMAE ProViCo VCL CORP FAME ST-Adapter TimeSformer
Builds on: Endo-FM DINO VideoMAE SimMIM TimeSformer
Closest papers by meaning
- Focus-to-Perceive Representation Learning: A Cognition-Inspired Hierarchical Framework for Endoscopic Video Analysis (CVPR 2026 · similarity 0.91)
- EndoSCOP-V: A Multi-Turn Video Understanding Evaluation Framework for Multimodal Models in Endoscopy Reporting and Clinical Reasoning (NeurIPS 2026 · similarity 0.87)
- EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysis (NeurIPS 2025 · similarity 0.85)
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.83)
- MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learning (CVPR 2024 · similarity 0.82)
- FOCUS: Knowledge-enhanced Adaptive Visual Compression for Few-shot Whole Slide Image Classification (CVPR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.