Research map / Vision-language and multimodal / CLIP and zero-shot
Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Cascade-CLIP splits CLIP's ViT-B/16 visual encoder into stages, aligns each stage's aggregated features with text embeddings via independent cascaded decoders, and fuses their masks, improving zero-shot semantic segmentation on COCO-Stuff, Pascal VOC and Pascal Context.
Problem
One-stage CLIP-based zero-shot semantic segmentation relies only on last-layer features, which lack object boundary detail; naively fusing intermediate CLIP layers breaks vision-language alignment and hurts unseen-class performance.
Approach
Split the CLIP visual encoder into stages, give each stage its own text-image decoder with stage-specific projected text embeddings, aggregate blocks within a stage via Neighborhood Gaussian Aggregation (trainable Gaussian block weights), and sum the masks across stages so intermediate features are aligned with text rather than directly fused.
Key results
- Cascade-CLIP improves unseen-class mIoU by 2.0% on COCO and 5.3% on Pascal VOC over prior state of the art in the inductive setting.
- Adding cascaded decoders to the ZegCLIP baseline improves unseen-class mIoU by 3.1%, with NGA adding further gains.
- Cascade-CLIP applied to ZegCLIP and SPT-SEG improves both methods' zero-shot results across benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Cascade-CLIP instead of the prior state-of-the-art method | zero-shot segmentation · COCO-Stuff | unseen-class mIoU: +2.0% | Helped |
| use Cascade-CLIP instead of the prior state-of-the-art method | zero-shot segmentation · Pascal-VOC | unseen-class mIoU: +5.3% | Helped |
| add cascaded decoders (ablation) | ZegCLIP · zero-shot segmentation | unseen-class mIoU: +3.1% | Helped |
| add Neighborhood Gaussian Aggregation (ablation) | ZegCLIP with cascaded decoders · zero-shot segmentation | unseen-class mIoU | Helped |
| directly fuse middle-layer and last-layer features (ablation) | ZegCLIP · zero-shot semantic segmentation | performance | Hurt |
| separate the last block into an independent stage (ablation) | Cascade-CLIP · zero-shot segmentation | performance | Helped |
| increase the number of cascaded decoders from 1 to 3 (ablation) | Cascade-CLIP · zero-shot segmentation | segmentation performance | Helped |
| reduce the number of blocks per stage from 3 to 2 (ablation) | Cascade-CLIP · zero-shot segmentation | performance | Hurt |
| use independent rather than shared text embeddings across stages (ablation) | Cascade-CLIP · zero-shot segmentation | mIoU | Helped |
| use learnable-weight NGA rather than common aggregation strategies (ablation) | Cascade-CLIP · zero-shot segmentation | performance | Helped |
| use multiple decoders on last-layer features rather than cascaded multi-level features (ablation) | multi-decoder model · zero-shot segmentation | performance | Hurt |
| directly fuse multi-layer features and use multiple decoders (ablation) | multi-decoder model · zero-shot segmentation | zero-shot capability | Hurt |
Surprising findings
- Directly fusing multi-layer CLIP features degrades zero-shot performance compared with last-layer-only baseline.
- Fusing features without independent cascaded decoders weakens CLIP's zero-shot ability on unseen classes even with multiple decoders.
Limitations
- Gains on seen classes and unseen classes depend on the chosen stage split and number of decoders, which were tuned empirically.
- Relies on frozen CLIP encoder with visual prompt tuning; generality beyond ViT-B/16 not shown in the main text.
Experimental setup
Datasets: COCO-Stuff Pascal VOC Pascal Context
Benchmarks: COCO-Stuff Pascal VOC Pascal Context
Metrics: mIoU seen mIoU unseen harmonic mIoU
Baselines: ZegCLIP ZegFormer Zsseg DeOP SPT-SEG ReCo CaR MaskCLIP+ FreeSeg GKC TCL
Builds on: ZegCLIP CLIP visual prompt tuning MMSegmentation
Closest papers by meaning
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.89)
- Exploring Regional Clues in CLIP for Zero-Shot Semantic Segmentation (CVPR 2024 · similarity 0.89)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.88)
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.88)
- A Simple Recipe for Language-guided Domain Generalized Segmentation (CVPR 2024 · similarity 0.87)
- Open-Vocabulary Semantic Segmentation with Image Embedding Balancing (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.