Research map / Detection and segmentation / CNN features
Toward Real Ultra Image Segmentation: Leveraging Surrounding Context to Cultivate General Segmentation Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SGNet, a plug-in surrounding-context branch on top of sliding-window inference that lets any general segmentation model handle ultra-high-resolution images with cross-patch consistency.
Problem
Ultra image segmentation methods are dataset-specific and cannot scale to arbitrary image sizes, while directly applying general segmentation models with sliding windows causes information bottleneck and fragmentation at patch borders.
Approach
A surrounding-context-guided branch takes a larger (2x) patch around each local patch, models its context with a surrounding context integration module (window attention plus GAP-based inter-window self-attention), and adds a boundary consistency L1 loss between overlapping predictions, so any encoder-decoder segmentation model can be plugged in.
Key results
- On DeepGlobe, DeepLabV3Plus+SGNet reaches 75.44% mIoU vs 73.22% baseline (+2.22), and all five general models gain between +1.69 and +2.90 on DeepGlobe.
- On Gleason, adding SGNet to FCN improves mIoU by +5.81 (52.65 to 58.46), and on Five-Billion-Pixels DeepLabV3Plus gains +5.76.
- A fast ISDNet-style variant reaches 74.28% mIoU on DeepGlobe at 25.59 FPS, above ISDNet's 73.30%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SGNet to DeepLabV3Plus | DeepLabV3Plus · DeepGlobe | mIoU: 73.22 -> 75.44 | Helped |
| Add SGNet to DeepLabV3Plus | DeepLabV3Plus · FBP | mIoU: 61.85 -> 63.18 | Helped |
| Add SGNet to DeepLabV3Plus | DeepLabV3Plus · Inria Aerial | mIoU: 80.88 -> 81.21 | Helped |
| Add SGNet to DeepLabV3Plus | DeepLabV3Plus · Gleason | mIoU: 55.45 -> 61.21 | Helped |
| Add SGNet to DeepLabV3Plus | DeepLabV3Plus · Cityscapes | mIoU: 75.24 -> 76.72 | Helped |
| Add surrounding-branch features (ablation) | DeepLabV3Plus · DeepGlobe | mIoU: 73.22 -> 73.76 | Helped |
| Add feature alignment (ablation) | DeepLabV3Plus with surrounding-branch features · DeepGlobe | mIoU: 73.76 -> 73.88 | Helped |
| Add SCI (ablation) | DeepLabV3Plus with feature alignment · DeepGlobe | mIoU: 73.88 -> 74.76 | Helped |
| Add an auxiliary segmentation head (ablation) | DeepLabV3Plus with SCI · DeepGlobe | mIoU: 73.88 -> 75.23 | Helped |
| Add boundary consistency loss (ablation) | DeepLabV3Plus with auxiliary head · DeepGlobe | mIoU: 75.23 -> 75.44 | Helped |
| Use GAP rather than convolution to extract the surrounding-window representation (ablation) | W-MSA context integration · DeepGlobe | mIoU: 75.17 -> 75.44 | Helped |
| Use late concatenation instead of early addition (ablation) | SGNet feature fusion · DeepGlobe | mIoU: 74.68 -> 75.44 | Helped |
Surprising findings
- Adding a naive trivial convolution-plus-transformer surrounding branch gives only +0.28 to +0.60 mIoU, far less than SCB, suggesting the specific design matters.
- Ensembling local and surrounding branch logits raises DeepLabV3Plus mIoU to 75.59%, suggesting part of the gain is ensembling.
Limitations
- Noise and artifacts in ultra-high-resolution images can hinder segmentation accuracy, as the authors note.
- Sliding inference with the full DeepLabV3Plus version is slow (0.66 FPS on DeepGlobe) compared with whole-image methods.
- No error bars or statistical significance tests are reported.
Experimental setup
Datasets: Cityscapes DeepGlobe Inria Aerial Five-Billion-Pixels Gleason CelebAMask-HQ
Benchmarks: Cityscapes DeepGlobe Inria Aerial Five-Billion-Pixels Gleason CelebAMask-HQ
Metrics: mIoU FPS GPU memory
Baselines: GLNet PPN MagNet FCtL ISDNet ElegantSeg WSDNet GPWFormer FCN DeepLabV3Plus HRNet SegFormer
Builds on: ISDNet GLNet DeepLabV3+ STDC
Closest papers by meaning
- Ultra-High Resolution Segmentation via Boundary-Enhanced Patch-Merging Transformer (AAAI 2025 · similarity 0.87)
- Uncertainty-aware Fine-tuning of Segmentation Foundation Models (NeurIPS 2024 · similarity 0.86)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.86)
- Unsupervised Universal Image Segmentation (CVPR 2024 · similarity 0.86)
- Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.86)
- Learning Hierarchical Image Segmentation For Recognition and By Recognition (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.