Research map / Detection and segmentation / Open-set and few-shot detection
Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Relationship Prompt Network (RPN), which uses a Relationship Prompt Module to make a frozen CLIP ViT-B/16 produce pixel-level OVSS predictions directly via prompt learning, with only about 3M trainable parameters and no segmentation-specific networks.
Problem
Open-vocabulary semantic segmentation typically needs extra segmentation-specific networks (mask proposal networks or semantic decoders) on top of VLMs, which makes training costly; prompt learning on VLMs has been limited to image-level granularity and lacks image-text relationships.
Approach
A Relationship Prompt Module generates pixel-level relationship prompts from image-text attention (image-to-pixel attention scheme) inserted layer-by-layer into the CLIP ViT encoder, combined with a two-linear-layer projection head, so the VLM itself outputs segmentation without segmentation-specific networks.
Key results
- In the zero-shot setting on VOC, RPN reaches 88.6 hIoU (mIoUu 84.6) without self-training, outperforming ZegCLIP's 84.3 hIoU.
- With only about 3.2M trainable parameters, RPN has lower FLOPs (84.2G vs 110.4G for ZegCLIP on VOC) and higher FPS (10.6 vs 9.0).
- In the open-vocabulary setting with ViT-L/14, RPN reaches 14.9 mIoU on A-847 and 36.4 on A-150, the best among compared methods on those datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RPN instead of the strongest listed baseline without self-training | CLIP ViT-B/16 · VOC zero-shot | mIoUu: 78.6 -> 84.6 | Helped |
| use RPN instead of the strongest listed baseline without self-training | CLIP ViT-B/16 · COCO zero-shot | mIoUu: 42.2 -> 42.8 | Helped |
| use RPN instead of the strongest listed baseline without self-training | CLIP ViT-B/16 · Context zero-shot | mIoUu: 54.6 -> 58.7 | Helped |
| add RPM to the CLIP with LPM baseline (ablation) | CLIP with LPM · VOC | hIoU: 59.6 -> 87.9 | Helped |
| add LPM to RPM (ablation) | RPN · VOC | hIoU: 86.3 -> 88.6 | Helped |
| use both image-level and pixel-level attention in ITP (ablation) | RPM · VOC | hIoU: 33.2 -> 88.6 | Helped |
| use trainable nn.Parameter instead of nn.Linear in APG (ablation) | RPM · VOC | hIoU: 86.0 -> 88.6 | Helped |
| increase M2oE dimension from r = 3 to r = 12 (ablation) | RPM · VOC | hIoU: 88.6 -> 88.8 | Helped |
| use M2oE instead of direct Multi-Scale aggregation (ablation) | RPM · VOC | hIoU: 88.2 -> 88.6 | Helped |
| use LPMc with Sigmoid loss instead of LPMc with Softmax loss (ablation) | RPN · VOC | hIoU: 71.4 -> 88.6 | Helped |
| use LPMc instead of LPMa with Sigmoid loss (ablation) | RPN · VOC | hIoU: 71.2 -> 88.6 | Helped |
| include the [CLS] token in the training-free projection module (TFPMc instead of TFPMa) (ablation) | RPN without trainable LPM layers · VOC | hIoU: 68.9 -> 86.7 | Helped |
Surprising findings
- Sigmoid-based losses (focal + dice) substantially outperform Softmax loss for the linear projection head, which the authors attribute to the relationship prompt following an image-to-pixel process.
- Pretrained weights matter: CLIP initialization gives much higher performance than ImageNet ViT or MAE, indicating the method still depends on large-scale multimodal pretraining.
Limitations
- Authors note that prompt learning could be improved by acting directly on the attention map, which they leave for future work.
- Performance is sensitive to the pretrained weights of the plain encoder; the method relies on CLIP's large-scale multimodal pretraining and does not work well with ViT or MAE initialization.
Experimental setup
Datasets: ADE20K Pascal VOC 2012 COCO-Stuff164K Pascal Context
Benchmarks: Pascal VOC 2012 COCO-Stuff164K Pascal Context ADE20K A-847 PC-459 A-150 PC-59 PAS-20
Metrics: pAcc mIoU seen mIoU unseen hIoU
Baselines: ZegFormer ZegCLIP FreeSeg ZSSeg MaskCLIP OVSeg CAT-Seg SAN SED FC-CLIP BitFit Adapter
Builds on: CLIP VPT LoRA ZegCLIP MaskCLIP
Closest papers by meaning
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.92)
- CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor (CVPR 2024 · similarity 0.90)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.89)
- Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.88)
- Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.