Research map / Detection and segmentation / Open-set and few-shot detection
Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes H-CLIP, a symmetric parameter-efficient fine-tuning of CLIP in hyperspherical space (block-diagonal transforms, orthogonality on the text encoder, and cross-layer/cross-modal tensor communication) for open-vocabulary semantic segmentation.
Problem
Fine-tuning CLIP for open-vocabulary semantic segmentation is costly, misaligns the image and text encoders, and degrades generalization to unseen categories.
Approach
Apply block-diagonal learnable transforms to both CLIP encoders symmetrically, enforce orthogonality (hyperspherical energy preservation) only on the text encoder, and add a dual cross-relation communication module built on tensor products to share information across layers and modalities.
Key results
- H-CLIP with ViT-L/14 reaches 16.5 A-847, 24.2 PC-459, 38.4 A-150, 64.1 PC-59, 97.7 PAS-20 mIoU, beating CAT-Seg on all listed sets.
- With ViT-B/16, H-CLIP trains about 5.6M parameters (around 4% of CLIP) and achieves 19.4 PC-459 and 32.4 A-150 mIoU.
- Ablation shows POP+DCRC together gives 12.5 A-847 vs 11.4 for LoRA-style baselines; DCRC alone adds only 0.01M params and limited gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use H-CLIP instead of CAT-Seg | CLIP ViT-B/16 · A-847 | mIoU: 12.0 -> 12.5 | Helped |
| Use H-CLIP instead of CAT-Seg | CLIP ViT-B/16 · PC-459 | mIoU: 19.0 -> 19.4 | Helped |
| Use H-CLIP instead of CAT-Seg | CLIP ViT-L/14 · PC-59 | mIoU: 63.3 -> 64.1 | Helped |
| Use H-CLIP instead of CAT-Seg | CLIP ViT-L/14 · PAS-20b | mIoU: 82.5 -> 83.2 | Helped |
| Add POP to the frozen baseline (ablation) | H-CLIP ViT-B/16 · A-847 | mIoU: 4.4 -> 12.3 | Helped |
| Add POP to the frozen baseline (ablation) | H-CLIP ViT-B/16 · PC-459 | mIoU: 6.6 -> 19.0 | Helped |
| Add DCRC to the frozen baseline (ablation) | H-CLIP ViT-B/16 · PC-459 | mIoU: 6.6 -> 10.9 | Helped |
| Add DCRC to POP (ablation) | H-CLIP ViT-B/16 · PC-459 | mIoU: 19.0 -> 19.4 | Helped |
| Use block dimension q=256 instead of q=128 (ablation) | H-CLIP ViT-B/16 POP · A-150 | mIoU: 32.4 -> 32.7 | Helped |
| Use block dimension q=64 instead of q=128 (ablation) | H-CLIP ViT-B/16 POP · A-847 | mIoU: 12.5 -> 11.7 | Hurt |
| Remove orthogonal constraints (ablation) | H-CLIP ViT-B/16 POP · A-847 | mIoU: 12.5 -> 11.9 | Hurt |
| Apply orthogonal constraints to all learnable matrices (ablation) | H-CLIP ViT-B/16 POP · A-150 | mIoU: 32.4 -> 31.4 | Hurt |
Surprising findings
- Applying orthogonal constraints to all layers, including the image encoder, or to no layers degrades performance on several datasets.
Limitations
- Orthogonal constraint on image encoder degrades performance, so asymmetric constraint design is needed.
- Evaluation limited to COCO-Stuff training and a single decoder (cost-aggregation decoder from CAT-Seg).
Experimental setup
Datasets: COCO-Stuff
Benchmarks: ADE20K-847 PASCAL-Context-459 ADE20K-150 PASCAL-Context-59 PASCAL VOC-20 PASCAL VOC-20b COCO (open-vocabulary detection)
Metrics: mIoU box AP50
Baselines: CAT-Seg SAN OVSeg FC-CLIP SED ZegFormer ZSseg ZegCLIP LoRA OFT VPT Adapter
Builds on: OFT CAT-Seg LoRA hyperspherical energy
Closest papers by meaning
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.87)
- SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images (CVPR 2025 · similarity 0.86)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.85)
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.85)
- Open-Vocabulary Segmentation with Semantic-Assisted Calibration (CVPR 2024 · similarity 0.85)
- Understanding Fine-tuning CLIP for Open-vocabulary Semantic Segmentation in Hyperbolic Space (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.