Research map / Vision-language and multimodal / CLIP and zero-shot
CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAT-Seg, which adapts CLIP for open-vocabulary semantic segmentation by aggregating the image-text cosine cost volume with spatial and class transformer aggregation and fine-tuning CLIP encoders (QV-only).
Problem
Open-vocabulary semantic segmentation requires adapting image-level CLIP to pixel-level labeling; prior end-to-end fine-tuning of CLIP overfits to seen classes, so most methods freeze the encoder and rely on region proposals.
Approach
Treat the cosine-similarity cost volume between CLIP dense image embeddings and text embeddings as the input to cost aggregation, using alternating spatial (Swin) and class (linear transformer) aggregation, so that fine-tuning CLIP encoders does not overfit to seen classes.
Key results
- Achieves 16.0 mIoU on A-847 and 23.8 mIoU on PC-459 with ViT-L/14, improving over the previous state of the art by +3.6 and +8.1 mIoU.
- With ViT-B/16 trained on COCO-Stuff, reaches 12.0 mIoU on A-847 and 19.0 on PC-459, surpassing prior methods using the same backbone.
- Inference is about 3.7x faster than ZegFormer-style region-text methods (0.54 s vs 2.70 s per image on RTX 3090).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace feature aggregation with cost aggregation and fine-tune CLIP (ablation) | Open-vocabulary segmentation model · A-847 | mIoU: 5.6 -> 14.7 | Helped |
| replace feature aggregation with cost aggregation and fine-tune CLIP (ablation) | Open-vocabulary segmentation model · PC-459 | mIoU: 12.8 -> 23.2 | Helped |
| add spatial and class aggregation to cost aggregation (ablation) | CAT-Seg · A-847 | mIoU: 14.7 -> 15.5 | Helped |
| add spatial and class aggregation to cost aggregation (ablation) | CAT-Seg · PC-459 | mIoU: 23.2 -> 23.2 | No clear effect |
| add embedding guidance (ablation) | CAT-Seg · A-847 | mIoU: 15.5 -> 16.0 | Helped |
| add embedding guidance (ablation) | CAT-Seg · PC-459 | mIoU: 23.2 -> 23.8 | Helped |
| use the upsampling decoder (ablation) | CAT-Seg · A-847 | mIoU: 9.9 -> 12.0 | Helped |
| fine-tune both CLIP encoders using query and value projections (ablation) | CAT-Seg · A-847 | mIoU: 10.4 -> 16.0 | Helped |
| fine-tune query and value projections in both encoders rather than all CLIP parameters (ablation) | CAT-Seg · A-847 | mIoU: 13.6 -> 16.0 | Helped |
| fine-tune query and value projections in both encoders rather than only the image encoder (ablation) | CAT-Seg · A-847 | mIoU: 13.9 -> 16.0 | Helped |
| fine-tune query and value projections in both encoders rather than only the text encoder (ablation) | CAT-Seg · A-847 | mIoU: 14.7 -> 16.0 | Helped |
| use CAT-Seg instead of SAN | CLIP ViT-L/14 segmentation model · MESS Medical Sciences | mIoU: 30.27 -> 24.70 | Hurt |
Surprising findings
- Feature aggregation with fine-tuning gains only marginally, whereas cost aggregation benefits substantially from fine-tuning CLIP, reversing the expected overfitting behavior of end-to-end fine-tuning.
Limitations
- Performance on medical sciences and engineering domains in MESS is inconsistent and near random for some settings.
- Relies on CLIP's limited knowledge in specialized domains, which the authors speculate causes weak medical results.
Experimental setup
Datasets: COCO-Stuff ADE20K PASCAL-Context PASCAL VOC MESS
Benchmarks: ADE20K (A-847, A-150) PASCAL-Context (PC-459, PC-59) PASCAL VOC (PAS-20, PAS-20b) MESS
Metrics: mIoU
Baselines: ZegFormer ZSseg OVSeg SAN FC-CLIP ODISE OpenSeg LSeg ZegCLIP X-Decoder OpenSeeD
Builds on: CLIP cost aggregation for visual correspondence FPN Swin Transformer
Closest papers by meaning
- Open-Vocabulary Semantic Segmentation with Image Embedding Balancing (CVPR 2024 · similarity 0.91)
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.91)
- SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.91)
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.90)
- Open-Vocabulary Segmentation with Semantic-Assisted Calibration (CVPR 2024 · similarity 0.90)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.