Research map / Vision-language and multimodal / CLIP and zero-shot
Transferable and Principled Efficiency for Open-Vocabulary Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Prunes the CLIP image encoder with semantic-agnostic iterative magnitude pruning (distillation loss only) to get transferable sparse subnetworks, and fine-tunes only layers with heavy-tailed-spectrum-based selection, for efficient open-vocabulary segmentation.
Problem
Open-vocabulary segmentation relies on large CLIP backbones and expensive fine-tuning, and existing compression and efficient fine-tuning methods are heuristic and must be re-run for each model or framework.
Approach
Prune the CLIP image encoder once with iterative magnitude pruning using only a CLIP knowledge-distillation loss (no semantic labels), so the sparse mask transfers to other OVS frameworks without customization; and pick which layers to fine-tune by a weight-spectrum power-law exponent (alpha) analysis that needs no data or forward/backward passes.
Key results
- Transferring the pruned subnetwork to FC-CLIP cuts parameters from 39.0M to 17.8M and FLOPs from 200.1G to 105.6G, with COCO mIoU 58.7 to 56.8 and ADE20K-150 23.3 to 19.1.
- Applying the heavy-tail layer selection (freezing 50% of layers with bottom alpha) to Han et al. raises ADE20K-150 mIoU from 16.6 to 17.3 and PC-59 from 39.0 to 39.9 while cutting training FLOPs by 12.0%.
- Combining subnetwork pruning with layer selection reduces training FLOPs of Han et al. to 32.6% of the original (122.2P vs 181.4P).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Transfer the semantic-agnostic pruned subnetwork to FC-CLIP | FC-CLIP · ADE20K-150 | mIoU: 23.3 -> 19.1 | Hurt |
| Transfer the semantic-agnostic pruned subnetwork to DeeplabV3 | DeeplabV3 · Cityscapes | mIoU: 20.3 -> 24.3 | Helped |
| Transfer the semantic-agnostic pruned subnetwork to Han et al. | Han et al. · Cityscapes | mIoU: 33.9 -> 31.7 | Hurt |
| Replace the learned semantic-agnostic pruning mask with a random subnetwork (ablation) | Han et al. · ADE20K-150 | mIoU: 15.8 -> 13.2 | Hurt |
| Freeze 50% of layers randomly during fine-tuning (ablation) | Han et al. · ADE20K-150 | mIoU: 16.6 -> 16.4 | Hurt |
| Freeze 50% of layers with top (greater) α values (ablation) | Han et al. · ADE20K-150 | mIoU: 16.6 -> 16.7 | Helped |
| Freeze 50% of layers with bottom (smaller) α values (ablation) | Han et al. · ADE20K-150 | mIoU: 16.6 -> 17.3 | Helped |
| Freeze 50% of layers with bottom (smaller) α values rather than top α values (ablation) | Han et al. · PC-59 | mIoU: 39.2 -> 39.9 | Helped |
| Combine selective layer-wise fine-tuning with subnetwork pruning (ablation) | Han et al. · COCO training | Training FLOPs: 181.4 P -> 122.2 P | Helped |
Surprising findings
- Freezing the layers with the smallest alpha (judged best-trained) and fine-tuning the rest improved mIoU over full fine-tuning, while freezing the layers with the largest alpha (not in the main method) was worse.
- Pruned DeepLabV3 subnetwork (19.1M params) outperforms the original dense DeepLabV3 on Cityscapes and ADE20K-150 after fine-tuning.
Limitations
- Evaluated only on ResNet-50 and convolutional backbones; ViT and larger backbones left for future work.
- Layer selection is coarse (layer-level), not weight- or channel-level.
- Performance gap to state-of-the-art Cat-Seg remains (about 6.4 mIoU on PAS-20 and 8.4 on ADE20K-150).
Experimental setup
Datasets: COCO-Panoptic ADE20K Cityscapes PASCAL VOC 2012 PASCAL Context
Benchmarks: COCO-Panoptic Cityscapes ADE20K-150 ADE20K-847 PASCAL VOC 2012 (PAS-20) PC-59 PC-459
Metrics: mIoU parameters FLOPs training FLOPs
Baselines: Han et al. DeepLabV3 FC-CLIP Random subnetwork Cat-Seg OVSeg ZegFormer Simbaseline OpenSeg LSeg
Builds on: Lottery Ticket Hypothesis Iterative Magnitude Pruning Han et al. OVS distillation framework FC-CLIP DeepLabV3 Mask2Former
Closest papers by meaning
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.91)
- OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentation (NeurIPS 2025 · similarity 0.90)
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.90)
- Renovating Names in Open-Vocabulary Segmentation Benchmarks (NeurIPS 2024 · similarity 0.89)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.89)
- Open-Vocabulary Segmentation with Semantic-Assisted Calibration (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.