Research map / Detection and segmentation / CNN features
EOV-Seg: Efficient Open-Vocabulary Panoptic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EOV-Seg, a single-stage shared open-vocabulary panoptic segmentation framework with a lightweight decoder, Vocabulary-Aware Selection, and Two-way Dynamic Embedding Experts that uses the first ViT-B CLIP block for spatial awareness, trading a little accuracy for large speed gains.
Problem
Open-vocabulary panoptic segmentation methods are computationally expensive: two-stage methods crop images repeatedly, and single-stage methods rely on heavyweight mask decoders to compensate for CNN-CLIP features lacking spatial location information.
Approach
Use a lightweight decoder on a shared CNN-CLIP backbone, make visual aggregated features vocabulary-aware via text-guided selection (VAS), and inject spatial information from the first block of ViT-B CLIP through a router-weighted fusion of mask and spatial embeddings (TDEE).
Key results
- With COCO-only training, EOV-Seg (L) achieves 24.5 PQ, 32.1 mIoU and 11.6 FPS on ADE20K open-vocabulary panoptic segmentation.
- EOV-Seg (S) with ResNet50 runs 23.8 FPS with 71M parameters on a single RTX 3090, reaching 15.1 PQ on ADE20K.
- EOV-Seg (L) is about 4x faster than FC-CLIP and 19x faster than ODISE while outperforming ODISE by 1.1 PQ on ADE20K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EOV-Seg (L) instead of FC-CLIP | ADE20K open-vocabulary panoptic segmentation · ADE20K | FPS: 3.0 -> 11.6 | Helped |
| Use EOV-Seg (L) instead of CAT-Seg | open-vocabulary semantic segmentation · ADE20K A-150 | mIoU: +2.0 mIoU | Helped |
| Use ViT-based CLIP block 0 instead of block 5 or 11 as spatial awareness extractor (ablation) | EOV-Seg · ADE20K | PQ: 23.8 -> 24.5; 23.7 -> 24.5 | Helped |
| Add SAM block 11 as spatial awareness extractor instead of ViT-based CLIP block 0 (ablation) | EOV-Seg · ADE20K | PQ: 24.5 -> 24.9 | Helped |
| Add VAS to TDEE-based EOV-Seg (ablation) | EOV-Seg · ADE20K | PQ: 23.6 -> 24.5 | Helped |
| Add VAS to EAF-based spatial fusion (ablation) | EOV-Seg with EAF · ADE20K | PQ: 23.0 -> 22.3 | Hurt |
| Add VAS to SDI-based spatial fusion (ablation) | EOV-Seg with SDI · ADE20K | PQ: 22.5 -> 22.9 | Helped |
| Replace EAF with TDEE and use VAS (ablation) | EOV-Seg · ADE20K | PQ: 22.3 -> 24.5 | Helped |
| Use single-scale instead of multi-scale features with dynamic depthwise attention (ablation) | EOV-Seg lightweight decoder · ADE20K | PQ: 22.7 -> 24.5 | Helped |
| Use geometric instead of arithmetic ensemble with α=0.4 and β=0.8 (ablation) | EOV-Seg (L) · ADE20K | PQ: 23.4 -> 24.5 | Helped |
Surprising findings
- Adding VAS to the early aggregation fusion variant decreased performance, which the authors attribute to changed spatial distributions of features.
- The first block of ViT-B CLIP matches SAM's last block in spatial awareness while being about 2.2x faster, and deeper CLIP blocks degrade results.
Limitations
- Accuracy is lower than the strongest heavyweight baselines on some benchmarks (e.g., EOV-Seg (S) 15.1 PQ vs. much larger models).
- Trained only on COCO Panoptic with a limited number of semantic classes.
Experimental setup
Datasets: COCO Panoptic
Benchmarks: ADE20K A-847 PC-459 A-150 PC-59 PAS-20 COCO Panoptic Cityscapes
Metrics: PQ SQ RQ AP mIoU FPS Params FLOPs
Baselines: ODISE FC-CLIP MasQCLIP OPSNet FreeSeg MaskCLIP CAT-Seg OVSeg SAN YOSO
Builds on: CLIP Mask2Former FC-CLIP modulated deformable convolution geometric ensemble
Closest papers by meaning
- OMG-Seg: Is One Model Good Enough For All Segmentation? (CVPR 2024 · similarity 0.88)
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.88)
- PanoGS: Gaussian-based Panoptic Segmentation for 3D Open Vocabulary Scene Understanding (CVPR 2025 · similarity 0.86)
- Open-Vocabulary Semantic Segmentation with Image Embedding Balancing (CVPR 2024 · similarity 0.86)
- Seg-VAR:Image Segmentation with Visual Autoregressive Modeling (NeurIPS 2025 · similarity 0.86)
- Leveraging Depth and Language for Open-Vocabulary Domain-Generalized Semantic Segmentation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.