Research map / Vision-language and multimodal / CLIP and zero-shot
Renovating Names in Open-Vocabulary Segmentation Benchmarks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RENOVATE, a scalable framework that automatically renames segments in open-vocabulary segmentation benchmarks (COCO, ADE20K, Cityscapes) using caption-derived context names, GPT-4 candidate generation, and a trained renaming model, improving training and evaluation.
Problem
Class names in open-vocabulary segmentation benchmarks are often inaccurate, too general, or lack context, hurting open-vocabulary training and evaluation; no scalable principled method existed for renaming them.
Approach
Generate per-class context nouns from image captions (CaSED), prompt GPT-4 for candidate names, then train a Mask2Former-style renaming model whose text-query decoder matches candidate names to ground-truth segments, assigning a renovated name per segment; also uses open metrics to credit semantically close misclassifications.
Key results
- Training FC-CLIP with RENOVATE names improves COCO PQ from 52.70 to 56.62 and ADE20K mIoU from 33.06 to 38.50 versus original names.
- Human preference study with 20 researchers favored RENOVATE names over original names in 82% of cases.
- Open metrics on ADE20K with FC-CLIP show a much larger standard-vs-open gap under RENOVATE names (PQ 27.90 standard vs 58.25 open).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train FC-CLIP with RENOVATE names instead of OpenSeg names | FC-CLIP · MS COCO | PQ: 53.60 -> 56.62 | Helped |
| Train FC-CLIP with RENOVATE names instead of Original names | FC-CLIP · MS COCO | mIoU: 60.91 -> 65.48 | Helped |
| Train FC-CLIP with RENOVATE names instead of OpenSeg names | FC-CLIP · ADE20K | mIoU: 35.63 -> 38.50 | Helped |
| Train FC-CLIP with RENOVATE names instead of OpenSeg names | FC-CLIP · Cityscapes | mIoU: 58.42 -> 61.25 | Helped |
| Add CaSED caption nouns as context names instead of using no context names (ablation) | RENOVATE candidate generation · ADE20K | PQ: 25.50 -> 27.90 | Helped |
| Use CaSED caption nouns rather than BLIP2 caption nouns as context names (ablation) | RENOVATE candidate generation · ADE20K | PQ: 25.89 -> 27.90 | Helped |
| Use CaSED caption nouns rather than RAM image tags as context names (ablation) | RENOVATE candidate generation · ADE20K | PQ: 26.95 -> 27.90 | Helped |
| Use text queries instead of learnable queries with a text classifier (ablation) | RENOVATE renaming model · ADE20K | PQ: 21.09 -> 25.65 | Helped |
| Randomly replace some ground-truth mask attention biases with predicted masks (ablation) | RENOVATE renaming model · ADE20K | PQ: 25.65 -> 26.21 | Helped |
| Append a negative name to the text queries (ablation) | RENOVATE renaming model · ADE20K | PQ: 26.21 -> 27.90 | Helped |
| Use negative sampling with RENOVATE names (ablation) | FC-CLIP training with RENOVATE names · Open-vocabulary segmentation | performance | Helped |
| Use negative sampling with Original names (ablation) | FC-CLIP training with Original names · Open-vocabulary segmentation | performance | No clear effect |
Surprising findings
- Synonym names and candidate names (without the renaming model) yield much worse training results than original names, e.g., Synonym names give COCO PQ 23.38 vs 52.70 for original names.
Limitations
- Renaming may propagate biases from foundation models (GPT-4, captioners) into new names.
- Performs poorly on very small objects and segments with insufficient visual cues; human verification needed for evaluation sets.
- Exploration limited to three panoptic datasets and one backbone family; scaling to large public datasets left for future work.
Experimental setup
Datasets: MS COCO ADE20K Cityscapes PMD
Benchmarks: MS COCO ADE20K Cityscapes
Metrics: PQ AP mIoU open PQ open AP open mIoU
Baselines: Original names OpenSeg names Synonym names Candidate names ODISE MasQCLIP
Builds on: Mask2Former FC-CLIP CaSED OpenSeg ODISE
Closest papers by meaning
- Transferable and Principled Efficiency for Open-Vocabulary Segmentation (CVPR 2024 · similarity 0.89)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.88)
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.87)
- Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation (CVPR 2025 · similarity 0.87)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.87)
- Open-Vocabulary Segmentation with Semantic-Assisted Calibration (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.