Research map / Video / Video segmentation and tracking
An Embedding-Unleashing Video Polyp Segmentation Framework via Region Linking and Scale Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an embedding-unleashing framework (PGN + AEN with cross-scale region linking and cross-wise scale alignment) that models video polyp segmentation as appearance-level semantic embedding, reaching SOTA on SUN-SEG and CVC-612 at real-time speed.
Problem
Video polyp segmentation in colonoscopy is hard due to low contrast between background and polyps and dramatic frame-to-frame variations of polyps; prior VPS methods model static and dynamic semantics at pixel level with dense features.
Approach
Model VPS as an appearance-level semantic embedding process: an auxiliary AEN produces region-linking embeddings (CRL) and polyp-center cross-wise alignment embeddings (CSA), which are fused into the mask proposals of a video segmentation network via a parameter-free dot-product interaction, removing the up-sampling predictor.
Key results
- On SUN-SEG-Easy, AEN + CFFM (Res2Net-50) reaches maxDice 0.810 and maxIoU 0.728 versus 0.772/0.692 for CFFM alone and 0.774 for PNS+.
- With HRNet-W48 backbone, AEN + CFFM reaches S-alpha 0.869 and maxDice 0.830 on SUN-SEG-Easy, and maxDice 0.882 on CVC-612.
- AEN + CFFM runs at 39 FPS with 33.81M parameters, versus 45 FPS for CFFM alone at 26.49M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add AEN to CFFM | CFFM with Res2Net-50 · SUN-SEG-Easy | maxDice: 0.772 -> 0.810 | Helped |
| Add AEN to CFFM | CFFM with Res2Net-50 · SUN-SEG-Hard | maxDice: 0.747 -> 0.786 | Helped |
| Add AEN to CFFM | CFFM with Res2Net-50 · CVC-612 | maxDice: 0.857 -> 0.878 | Helped |
| Add CRL (ablation) | AEN + CFFM · SUN-SEG-Easy | maxDice: 0.793 -> 0.810 | Helped |
| Add CRL (ablation) | AEN + CFFM · SUN-SEG-Hard | maxDice: 0.774 -> 0.786 | Helped |
| Add CSA (ablation) | AEN + CFFM · SUN-SEG-Easy | maxDice: 0.786 -> 0.810 | Helped |
| Add CSA (ablation) | AEN + CFFM · SUN-SEG-Hard | maxDice: 0.766 -> 0.786 | Helped |
| Use frame-wise and clip-wise alignment instead of no alignment (ablation) | AEN + CFFM · SUN-SEG-Easy | maxDice: 0.790 -> 0.810 | Helped |
| Use frame-wise and clip-wise alignment instead of no alignment (ablation) | AEN + CFFM · SUN-SEG-Hard | maxDice: 0.771 -> 0.786 | Helped |
Limitations
- Fails on facula interference (reflective highlights) in colonoscopy frames
- Struggles with small low-contrast polyps
- Struggles with polyps of dramatic shape
Experimental setup
Datasets: SUN-SEG CVC-612
Benchmarks: SUN-SEG-Easy SUN-SEG-Hard CVC-612
Metrics: S-alpha (structure measure) maxDice maxIoU FPS parameters GFLOPs
Baselines: PraNet FSNet PNSNet PNS+ CFFM GenVIS META-UNet PIDNet
Builds on: CFFM PNS+ GenVIS Mask2Former-style pixel decoder
Closest papers by meaning
- HFSTI-Net: Hierarchical Frequency-spatial-temporal Interactions for Video Polyp Segmentation (ICLR 2026 · similarity 0.90)
- VPSentry: Semi-supervised Video Polyp Segmentation via Sentry-guided Long-term Prototype Fusion with Correlation Dynamic Propagation (AAAI 2026 · similarity 0.90)
- Boost the Inference with Co-training: A Depth-guided Mutual Learning Framework for Semi-supervised Medical Polyp Segmentation (CVPR 2025 · similarity 0.84)
- Polyper: Boundary Sensitive Polyp Segmentation (AAAI 2024 · similarity 0.84)
- Why Invariance is Not Enough for Biomedical Domain Generalization and How to Fix It (NeurIPS 2026 · similarity 0.81)
- Multi-view Masked Contrastive Representation Learning for Endoscopic Video Analysis (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.