Research map / Detection and segmentation / Open-set and few-shot detection
Open-Vocabulary Semantic Segmentation with Image Embedding Balancing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EBSeg, an open-vocabulary semantic segmentation framework that adaptively balances three image embeddings from an AdaB Decoder, aligns inter-class affinity with CLIP text space via an SSC loss, and fuses frozen SAM features with CLIP features.
Problem
Open-vocabulary semantic segmentation models built on CLIP overfit to training classes seen in segmentation datasets, hurting generalization to new classes; CLIP features also lack spatial detail due to low resolution and image-level supervision.
Approach
An AdaB Decoder produces three image embeddings (fully supervised, mask-attention from CLIP, and frozen CLIP mask-pooled) that are combined with different preset weights for training versus new classes; an SSC loss distills CLIP text inter-class similarity structure into image embeddings; frozen SAM encoder features are added to CLIP features for spatial detail.
Key results
- With CLIP ViT-B/16, EBSeg reaches 11.1 mIoU on A-847, 17.3 on PC-459, 30.0 on A-150, 56.7 on PC-59 and 94.6 on VOC, outperforming SAN on all five.
- With CLIP ViT-L/14, EBSeg reaches 13.7 mIoU on A-847, 21.0 on PC-459, 32.8 on A-150, 60.2 on PC-59 and 96.4 on VOC.
- On ADE20K-150 ablation, the AdaB Decoder raises new-class mIoU from 20.5 to 21.6 and SSC loss raises it from 19.9 to 21.6 relative to ablated versions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EBSeg instead of SAN with CLIP ViT-B/16 | open-vocabulary semantic segmentation · Pascal Context-459 | mIoU: 12.6 -> 17.3 | Helped |
| Use EBSeg instead of SAN with CLIP ViT-L/14 | open-vocabulary semantic segmentation · Pascal Context-459 | mIoU: 15.7 -> 21.0 | Helped |
| Add the AdaB Decoder and SSC Loss (ablation) | EBSeg · ADE20K-150 | mIoU: 29.2 -> 30.0 | Helped |
| Balance A, B, and D image embeddings instead of using B alone (ablation) | EBSeg AdaB Decoder · ADE20K-150 | mIoU: 29.0 -> 30.0 | Helped |
| Add adaptive balancing of image embeddings (ablation) | EBSeg · ADE20K-150 validation training classes | mIoU train: 39.5 -> 40.3(+0.8) | Helped |
| Add adaptive balancing of image embeddings (ablation) | EBSeg · ADE20K-150 validation new classes | mIoU new: 20.5 -> 21.6(+1.1) | Helped |
| Apply SSC Loss to A embeddings from all decoder layers (ablation) | EBSeg · ADE20K-150 | mIoU: 29.2 -> 28.6 | Hurt |
| Apply SSC Loss to A embeddings from the last decoder layer only (ablation) | EBSeg · ADE20K-150 | mIoU: 29.2 -> 30.0 | Helped |
| Add SSC Loss (ablation) | EBSeg · ADE20K-150 validation new classes | mIoU new: 19.9 -> 21.6(+1.7) | Helped |
Surprising findings
- Applying the SSC loss to mask attention embeddings B, or to both A and B, hurt performance relative to applying it to the last-layer embeddings A only.
Limitations
- Balancing weights (alpha, beta, gamma) are manually preset rather than learned; authors note learnable weights as future work.
- SSC loss on mask attention embeddings B is difficult to optimize because many parameters are frozen.
- Performance on training classes drops slightly with the SSC loss (40.6 to 40.3 mIoU on training classes).
Experimental setup
Datasets: COCO-Stuff
Benchmarks: ADE20K-847 Pascal Context-459 ADE20K-150 Pascal Context-59 Pascal VOC
Metrics: mIoU
Baselines: ZegFormer SimSeg OVSeg SAN ODISE MaskCLIP OpenSeg LSeg+
Builds on: Mask2Former SAN MaskCLIP STEGO CLIP Surgery
Closest papers by meaning
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.91)
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.91)
- SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.91)
- USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.90)
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.90)
- Understanding Fine-tuning CLIP for Open-vocabulary Semantic Segmentation in Hyperbolic Space (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.