Research map / Vision-language and multimodal / CLIP and zero-shot
Understanding Fine-tuning CLIP for Open-vocabulary Semantic Segmentation in Hyperbolic Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyperCLIP, which fine-tunes CLIP's text and image encoders in the Poincaré ball by learning block-diagonal scaling matrices (via Möbius matrix multiplication) that shrink embedding hyperbolic radius, reaching SOTA open-vocabulary segmentation with ~4% of CLIP's parameters tuned.
Problem
Explaining why jointly fine-tuning CLIP's text and image encoders improves open-vocabulary semantic segmentation, and designing a parameter-efficient fine-tuning method that adjusts text embeddings to the pixel-level hierarchy.
Approach
Interpret the benefit of text-encoder fine-tuning as shrinking the hyperbolic radius of CLIP text embeddings to match pixel-level hierarchy, and implement this with learnable block-diagonal scaling matrices applied via Möbius matrix multiplication in the Poincaré ball.
Key results
- HyperCLIP with ViT-B/16 reaches 12.3 mIoU on A-847 and 19.2 on PC-459, outperforming CAT-Seg (12.0, 19.0) and SAN (10.1, 12.6).
- With ViT-L/14, HyperCLIP achieves 16.3 on A-847, 24.1 on PC-459, 98.3 on PAS-20, versus CAT-Seg's 16.0, 23.8, 97.0.
- Fine-tunes only about 5.6M parameters (about 4% of CLIP) on ViT-B/16, fewer than CAT-Seg (39.5M) and SAN (8.4M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CLIP in hyperbolic space with radius-adjusting scaling matrices | CLIP ViT-B/16 · ADE20K A-847 | mIoU: 12.0 -> 12.3 | Helped |
| Fine-tune CLIP in hyperbolic space with radius-adjusting scaling matrices | CLIP ViT-L/14 · PASCAL-Context PC-59 | mIoU: 63.3 -> 64.2 | Helped |
| Fine-tune CLIP in hyperbolic space with radius-adjusting scaling matrices | CLIP · COCO open-vocabulary object detection | APBase50: 25.2 -> 26.3 | Helped |
| Fine-tune CLIP in hyperbolic space with radius-adjusting scaling matrices | CLIP · ADE20K open-vocabulary panoptic segmentation | PQ: 27.1 -> 29.2 | Helped |
| Increase diagonal block size from 1 to 128 (ablation) | HyperCLIP, CLIP ViT-B/16 · ADE20K A-847 | mIoU: 5.9 -> 12.3 | Helped |
| Increase diagonal block size beyond 128 to 256 (ablation) | HyperCLIP, CLIP ViT-B/16 · ADE20K A-847 | mIoU: 12.3 -> 12.2 | No clear effect |
| Fine-tune both image and text encoders rather than only the text encoder (ablation) | HyperCLIP, CLIP ViT-B/16 · ADE20K A-847 | mIoU: 11.1 -> 12.3 | Helped |
| Fine-tune both image and text encoders rather than only the image encoder (ablation) | HyperCLIP, CLIP ViT-B/16 · ADE20K A-847 | mIoU: 10.3 -> 12.3 | Helped |
| Replace standard matrix multiplication with Möbius matrix multiplication (ablation) | HyperCLIP, CLIP ViT-B/16 · ADE20K A-847 | mIoU: 11.1 -> 12.3 | Helped |
Surprising findings
- Fine-tuning only one encoder leaves one modality at image level and falls short of joint fine-tuning, suggesting alignment at the same hierarchical level is needed.
- Controlling the average hyperbolic radius of text embeddings to around 6.0 gives the best segmentation, with performance dropping when the target is raised or lowered.
Limitations
- Explanation of the hyperbolic radius as the granularity measure is supported by empirical observation rather than a formal theory of segmentation granularity.
- Evaluation of the hyperbolic radius hypothesis is limited to CLIP ViT-B/16 and COCO-Stuff training.
Experimental setup
Datasets: COCO-Stuff
Benchmarks: ADE20K A-847 ADE20K A-150 PASCAL-Context PC-459 PASCAL-Context PC-59 PASCAL VOC PAS-20 PASCAL VOC PAS-20b COCO open-vocabulary object detection ADE20K open-vocabulary panoptic segmentation
Metrics: mIoU AP50 PQ
Baselines: CAT-Seg SAN OVSeg ZegCLIP FC-CLIP SED ODISE LoRA HyperLoRA CLIM MAFT+
Builds on: CAT-Seg HyperLoRA LoRA gyrovector spaces
Closest papers by meaning
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.90)
- Open-Vocabulary Semantic Segmentation with Image Embedding Balancing (CVPR 2024 · similarity 0.90)
- Open-Vocabulary Segmentation with Semantic-Assisted Calibration (CVPR 2024 · similarity 0.89)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.89)
- Compositional Entailment Learning for Hyperbolic Vision-Language Models (ICLR 2025 · similarity 0.89)
- Dual Semantic Guidance for Open Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.