Research map / Detection and segmentation / Weakly supervised detection and segmentation
Active Label Correction for Semantic Segmentation with Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes active label correction (ALC) for semantic segmentation that starts from Grounded-SAM pseudo labels, uses a correction query (verify-then-relabel), superpixel-based diversified pixel pool, and a look-ahead SIM acquisition with label expansion, and releases the corrected PASCAL+ dataset.
Problem
Pixel-wise annotation for semantic segmentation is labor-intensive and error-prone, and zero-shot foundation model pseudo labels are too noisy to train or validate models reliably; standard active learning suffers a cold-start problem and costly pixel-wise queries.
Approach
Ask annotators a correction query (confirm the foundation-model pseudo label, and only give the correct label if it is wrong), select pixels from a superpixel-based diversified pool, and use a look-ahead acquisition (SIM) that weights uncertainty by cosine similarity to propagate corrections across each superpixel.
Key results
- ALC reaches 95% of fully supervised PASCAL performance with about 6K clicks, 30% of the budget of the prior SOTA; on Cityscapes it uses 75% of the prior budget (150K clicks).
- In a 20-annotator user study, the correction query took 4.76 s per query versus 6.31 s for the classification query at equal 95% accuracy.
- Correcting PASCAL with the framework changed 2.6 million pixels (0.5%), raising training-set mIoU by about 0.9% and producing the PASCAL+ dataset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SIM instead of LCIL for pixel acquisition | ALC · PASCAL, 5K clicks | Model mIoU: 56.82$_{\pm 0.05}$ -> 68.72$_{\pm 0.10}$ | Helped |
| Use the correction query instead of the classification query | ALC annotation · User study | Time per query (s): 6.31$_{\pm 0.99}$ -> 4.76$_{\pm 0.45}$ | Helped |
| Use the correction query instead of the classification query | ALC annotation · User study | Accuracy (%): 95.0$_{\pm 3.3}$ -> 95.0$_{\pm 4.0}$ | No clear effect |
| Correct PASCAL training labels to create PASCAL+ | DeepLab-v3+ ResNet101 trained on PASCAL · P validation | Model mIoU (%): 75.36$_{\pm 0.07}$ -> 75.78$_{\pm 0.12}$ | Helped |
| Use both corrected train and validation labels (PASCAL+) | DeepLab-v3+ ResNet101 · PASCAL+ train and validation | Model mIoU (%): 75.36$_{\pm 0.07}$ -> 76.42$_{\pm 0.03}$ | Helped |
| Add look-ahead acquisition (ablation) | ALC · PASCAL, 5K budget | Model mIoU: 56.82$_{\pm 0.05}$ -> 68.72$_{\pm 0.10}$ | Helped |
| Add label expansion (ablation) | ALC · PASCAL, 5K budget | Model mIoU: 56.69$_{\pm 0.35}$ -> 68.72$_{\pm 0.10}$ | Helped |
| Add a diversified pixel pool (ablation) | ALC · PASCAL, 5K budget | Model mIoU: 56.01$_{\pm 0.58}$ -> 68.72$_{\pm 0.10}$ | Helped |
| Use SAM superpixels instead of SEEDS with Spx warm-start (ablation) | Spx · PASCAL, 3K clicks | Model mIoU (%): 57.77$_{\pm 0.70}$ -> 57.79$_{\pm 0.66}$ | No clear effect |
| Warm-start Spx with Grounded-SAM pseudo-labels instead of cold-start (ablation) | Spx with SEEDS superpixels · PASCAL, 3K clicks | Model mIoU (%): 52.34$_{\pm 0.85}$ -> 57.77$_{\pm 0.70}$ | Helped |
| Set label-expansion similarity threshold to 1.0 instead of 0.0 (ablation) | ALC · PASCAL, 5K pixels | Model mIoU: $68.71$ -> $56.05$ | Hurt |
| Use the superpixel-based diversified pixel pool instead of PixelPick (ablation) | ALC with SIM, look-ahead, and label expansion · PASCAL, 5K budget | Model mIoU: 62.59 -> 68.71 | Helped |
Surprising findings
- Lower Grounded-SAM box thresholds give more objects but noisier pseudo labels, so initial pseudo-label mIoU does not improve monotonically.
- Adding SAM superpixels alone to the Spx baseline gives negligible gain, and the warm-start gain over cold-start for Spx is small; most gains come from the SIM acquisition.
Limitations
- Depends on foundation models (Grounded-SAM) and inherits their failure modes, such as incomplete superpixels for minor domains.
- Cost savings from the correction query rely on the assumed pseudo-label accuracy p (0.27 on PASCAL, 0.5 on Cityscapes), and user-study results are from a small setup of 20 queries per annotator.
Experimental setup
Datasets: PASCAL VOC 2012 Cityscapes Kvasir-SEG ImageNet
Benchmarks: PASCAL VOC 2012 Cityscapes Kvasir-SEG
Metrics: mIoU precision recall clicks to reach 95% of fully supervised performance annotation time
Baselines: Spx MerSpx MulSpx LCIL SoftMin AIoU PixelPick Entropy BvSB ClassBal
Builds on: Grounded-SAM Confident Learning DeepLab-v3+ SEEDS superpixels PixelPick
Closest papers by meaning
- A²LC: Active and Automated Label Correction for Semantic Segmentation (AAAI 2026 · similarity 0.95)
- Diffusion-Driven Two-Stage Active Learning for Low-Budget Semantic Segmentation (NeurIPS 2025 · similarity 0.88)
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.87)
- Pixels to Tokens: Token Space Efficient Active Learning for Low-Budget Semantic Segmentation (NeurIPS 2026 · similarity 0.87)
- Connectivity-Driven Pseudo-Labeling Makes Stronger Cross-Domain Segmenters (NeurIPS 2024 · similarity 0.86)
- AdaCo: Overcoming Visual Foundation Model Noise in 3D Semantic Segmentation via Adaptive Label Correction (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.