Research map / Detection and segmentation / Open-set and few-shot detection
Open-vocabulary Panoptic Segmentation with Embedding Modulation
Xi Chen, Shuang Li, Ser-Nam Lim, Antonio Torralba, Hengshuang Zhao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add mask filtering using classification score times IoU score to CLIP-embedding recognition (ablation) | CAG-Seg + CLIP Embeds · ADE20K panoptic | PQ: 4.9 -> 10.7 | Improved |
| use query embeddings instead of CLIP embeddings for recognition (ablation) | CAG-Seg · COCO panoptic | PQ: 22.7 -> 51.9 | Improved |
| add Embedding Modulation (fusion and logits debiasing) (ablation) | OPSNet · ADE20K panoptic | PQ: 14.6 -> 17.7 | Improved |
| add logits debiasing on top of embedding fusion (ablation) | OPSNet · ADE20K panoptic | PQ: 16.9 -> 17.7 | Improved |
| naive ensemble with more CLIP weight (ablation) | query embeddings · COCO vs ADE20K panoptic | PQ: COCO 51.4 -> 43.8; ADE20K 16.4 -> 18.1 (peak at 3x) | Mixed |
| mask-pooling with one CLIP pass instead of masking or cropping each object (ablation) | CLIP embedding extraction · ADE20K panoptic | PQ: 9.4 -> 10.7 | Improved |
| use class score times IoU for mask filtering (ablation) | query embeddings · ADE20K panoptic | PQ: 13.6 -> 14.5 | Improved |
| add cross-attention to gather CLIP features into query embeddings (ablation) | CAG-Seg + Query Embeds · ADE20K panoptic | PQ: 14.5 -> 14.6 | No clear effect |
| add ImageNet-Val image-level classification supervision (ablation) | OPSNet ResNet-50 · COCO panoptic | PQ: 52.4 -> 51.3 | Hurt |
| add ImageNet classification and mask supervision (decoupled supervision) (ablation) | OPSNet ResNet-50 · ADE20K panoptic | PQ: 17.7 -> 19.0 | Improved |
| OPSNet versus Mask2Former on closed-vocabulary COCO | Mask2Former (Swin-L) · COCO panoptic val | PQ: 57.8 -> 57.9 | Improved |
Closest papers by meaning
- Open-Vocabulary Panoptic Segmentation With Text-to-Image Diffusion Models (CVPR 2023 · similarity 0.92)
- USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.90)
- Open-Vocabulary Semantic Segmentation with Image Embedding Balancing (CVPR 2024 · similarity 0.89)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.89)
- EOV-Seg: Efficient Open-Vocabulary Panoptic Segmentation (AAAI 2025 · similarity 0.89)
- SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
