Research map / Vision-language and multimodal / Cross-modal learning
Multi-modal Prototypes for Open-World Semantic Segmentation
Yuhuan Yang, Chaofan Ma, Chen Ju, Fei Zhang, Jiangchao Yao, Ya Zhang, Yanfeng Wang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text guidance to visual prototypes (full vs image only) | Multi-modal prototype model · COCO-20i Z/FS (mean of 4 folds) | mIoU: 43.4 -> 45.7 | Improved |
| add visual guidance to text-only model | Multi-modal prototype model · COCO-20i Z/FS (mean of 4 folds) | mIoU: 26.9 -> 45.7 | Improved |
| add text to image-only model | Multi-modal prototype model · PASCAL-5i Z/FS (mean of 4 folds) | mIoU: 63.9 -> 65.5 | Improved |
| multi-modal GFS vs DIaM | Ours vs DIaM · PASCAL-5i GFS 1-shot | UnSeen mIoU: 35.11 -> 39.44 | Improved |
| multi-modal GFS vs DIaM | Ours vs DIaM · COCO-20i GFS 1-shot | UnSeen mIoU: 17.22 -> 19.48 | Improved |
| increase visual prototypes from 1 to 5 (ablation) | Multi-modal prototype model · PASCAL-5i GFS ablation | HIoU: 39.39 -> 45.39 | Improved |
| decompose class names into 3 descriptions instead of 1 (ablation) | Multi-modal prototype model · PASCAL-5i GFS ablation | HIoU: 42.37 -> 44.17 | Improved |
| increase descriptions from 3 to 5 (ablation) | Multi-modal prototype model · PASCAL-5i GFS ablation | HIoU: 44.17 -> 42.89 | Hurt |
| add L=2 feature level to L=1 (ablation) | Multi-modal prototype model · PASCAL-5i GFS ablation | HIoU: 31.48 -> 39.99 | Improved |
| add L=3 feature level to L=1+2 (ablation) | Multi-modal prototype model · PASCAL-5i GFS ablation | HIoU: 39.99 -> 44.17 | Improved |
| use mask M-splitting instead of K-means for prototypes (ablation) | Multi-modal prototype model · Mask splitting speed | seconds per image: 22.9 -> 0.043 | Improved |
| use mask instead of box annotation for support (ablation) | Multi-modal prototype model · PASCAL-5i GFS ablation | HIoU: 35.00 -> 44.17 | Improved |
Closest papers by meaning
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.90)
- OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation (NeurIPS 2025 · similarity 0.89)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.89)
- Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP (AAAI 2025 · similarity 0.89)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.89)
- Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
