Research map / Vision-language and multimodal / CLIP and zero-shot
Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Image-Text Co-Decomposition (CoDe), which decomposes paired images and texts into region and word segments and aligns them contrastively, with learnable region/word highlighting prompts, for text-supervised semantic segmentation trained on CC3M and CC12M.
Problem
Text-supervised semantic segmentation learns from image-text pairs without dense labels, but image-text and region-text alignment mismatch semantic segmentation because a caption often contains multiple concepts while segments should be semantically homogeneous.
Approach
Jointly decompose the image into object regions and the text into word segments for a selected noun, align them with contrastive loss (region-word alignment), and use learnable prompts to fill masked-out image/text areas so CLIP features on highlighted segments are not corrupted by blank tokens.
Key results
- CoDe achieves average mIoU of 31.8 across six benchmarks, versus 30.1 for TCL, the strongest prior method in the table.
- CoDe is best on all six datasets, e.g., 57.7 on PASCAL VOC and 17.7 on ADE20K.
- Ablation: co-decomposition alone raises the average from 29.8 to 31.0; adding word and region highlighting prompts reaches 31.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add image-text co-decomposition (ablation) | Baseline text-supervised segmenter · Six-dataset average | mIoU: 29.8 -> 31.0 | Helped |
| Add a word highlighting prompt to the co-decomposition model (ablation) | Co-decomposition model · Six-dataset average | mIoU: 31.0 -> 31.3 | Helped |
| Add a region highlighting prompt to the co-decomposition model with word highlighting (ablation) | Co-decomposition model · Six-dataset average | mIoU: 31.3 -> 31.8 | Helped |
| Set the highlighted region-word contrastive loss weight λhcl to 0.05 instead of 0.5 (ablation) | CoDe · Six-dataset average | mIoU: 31.8 -> 30.6 | Hurt |
| Set λhcl to 0.1 instead of 0.5 (ablation) | CoDe · Six-dataset average | mIoU: 31.8 -> 31.2 | Hurt |
| Set λhcl to 0.25 instead of 0.5 (ablation) | CoDe · Six-dataset average | mIoU: 31.8 -> 31.7 | Hurt |
| Set λhcl to 0.75 instead of 0.5 (ablation) | CoDe · Six-dataset average | mIoU: 31.8 -> 31.5 | Hurt |
| Set λhcl to 1.0 instead of 0.5 (ablation) | CoDe · Six-dataset average | mIoU: 31.8 -> 30.8 | Hurt |
| Replace word-segment embeddings with extracted noun embeddings in the contrastive loss (ablation) | CoDe · Six-dataset average | mIoU: 31.8 -> 30.2% | Hurt |
| Use region-word co-decomposition and alignment | CoDe · Six zero-shot semantic segmentation benchmarks | mIoU: +1.7 mIoU | Helped |
Limitations
- Baseline numbers are taken from original papers under differing training data mixtures, so comparisons are not controlled.
- Only three of the nine compared methods were evaluated on the without-background datasets.
Experimental setup
Datasets: CC3M CC12M
Benchmarks: PASCAL VOC PASCAL Context COCO-Object COCO-Stuff Cityscapes ADE20K
Metrics: mIoU
Baselines: GroupViT ViL-Seg ViewCo OVSegmentor SimSeg TCL SegCLIP CoCu PGSeg
Builds on: TCL CLIP KgCoOp CoOp GroupViT
Closest papers by meaning
- Separate and Conquer: Decoupling Co-occurrence via Decomposition and Representation for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.88)
- Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIP (AAAI 2025 · similarity 0.87)
- Exploring CLIP's Dense Knowledge for Weakly Supervised Semantic Segmentation (CVPR 2025 · similarity 0.87)
- Text4Seg: Reimagining Image Segmentation as Text Generation (ICLR 2025 · similarity 0.86)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.86)
- GOAL: Global-local Object Alignment Learning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.