Research map / Detection and segmentation / Open-set and few-shot detection
Excluding the Impossible for Open Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ELSE-Net, which uses reverse "exclusion" probabilities from a fine-tuned CLIPN text encoder and a Reverse Retrieval Adapter to prune improbable categories from CLIP-based mask classification for open-vocabulary semantic segmentation.
Problem
Forward CLIP-based open-vocabulary semantic segmentation methods are biased toward recognizing underrepresented or novel categories as common ones, leading to misclassification.
Approach
Adds a model-agnostic EXP-Block that predicts the probability a mask does NOT belong to a class (via reverse prompts "A photo without [CLASSNAME]" with a tunable CLIPN text encoder and an R2-Adapter) and zeroes out inclusion probabilities above a threshold.
Key results
- ELSE-Net reaches 95.1 mIoU on Pascal VOC, versus 94.0 for the prior SOTA SAN, and is best on ADE-847 (10.3 vs 10.1) and PC-59 (54.1 vs 53.8).
- Ablation on VOC: adding the EXP-Block raises mIoU from 89.6 to 95.1 (about 5.5 points).
- Plugging the EXP-Block into SAN, FreeSeg and OVSeg improves mIoU by 0.0 to 4.4 points over reproduced baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the EXP-Block (ablation) | ELSE-Net · VOC | mIOU: 89.6 -> 95.1 | Helped |
| fine-tune rather than freeze the CLIPN Text Encoder (ablation) | ELSE-Net · VOC | mIOU: 2.2 points | Helped |
| add the R2-Adapter (ablation) | ELSE-Net · VOC | mIOU: 0.2 mIOU | Helped |
| use probability correction instead of Pin × (1 − Pex) (ablation) | ELSE-Net · VOC | mIOU: 1.3 points | Helped |
| add the EXP-Block | SAN · ADE-847 | mIOU: 7.9 -> 10.1 | Helped |
| add the EXP-Block | SAN · ADE-150 | mIOU: 26.9 -> 27.4 | Helped |
| add the EXP-Block | SAN · PC-59 | mIOU: 53.2 -> 53.7 | Helped |
| add the EXP-Block | SAN · VOC | mIOU: 94.2 -> 94.2 | No clear effect |
| add the EXP-Block | FreeSeg · ADE-150 | mIOU: 25.6 -> 27.6 | Helped |
| add the EXP-Block | FreeSeg · VOC | mIOU: 80.2 -> 82.7 | Helped |
| add the EXP-Block | OVSeg · ADE-847 | mIOU: 5.5 -> 8.3 | Helped |
| add the EXP-Block | OVSeg · ADE-150 | mIOU: 25.6 -> 26.4 | Helped |
Surprising findings
- The EXP-Block gives no gain on VOC for SAN reproduction (+0.0), and on ADE-150 ELSE-Net is second-best rather than best.
Limitations
- Exclusion sometimes is ineffective and offers limited benefit to the forward process.
- The off-the-shelf CLIPN is not fully compatible with the forward model; a tailored retrained model is needed.
- Relative gains on ADE-150 and other datasets are small, and the threshold and adapter weight were hand-set.
Experimental setup
Datasets: COCO-Stuff
Benchmarks: Pascal VOC ADE20K-847 ADE20K-150 Pascal Context-59
Metrics: mIoU
Baselines: LSeg+ Simseg FreeSeg DeOp OVSeg SAN GKC MaskCLIP SPT-SEG
Builds on: CLIP CLIPN SAN OVSeg DeOp
Closest papers by meaning
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.89)
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.88)
- Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation (CVPR 2025 · similarity 0.88)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.88)
- Open-Vocabulary Segmentation with Semantic-Assisted Calibration (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.