Research map / Detection and segmentation / CNN features
SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SED, a single-stage encoder-decoder for open-vocabulary semantic segmentation that uses a hierarchical ConvNeXt CLIP encoder for cost maps, a gradual fusion decoder, and top-k category early rejection to speed up inference.
Problem
Open-vocabulary semantic segmentation with CLIP-based single-stage methods suffers from weak local spatial information and quadratic cost of plain ViT backbones, and decoder cost grows with the number of categories, limiting speed and accuracy.
Approach
Replace the plain ViT image encoder in cost-map methods (e.g., CAT-Seg) with a hierarchical ConvNeXt CLIP encoder that provides multi-scale features for skip-layer fusion, add a gradual fusion decoder with large-kernel and class-level aggregation, and prune non-existing categories early in the decoder via top-k predictions from auxiliary heads during inference.
Key results
- With ConvNeXt-B, SED reaches 31.6% mIoU on A-150 at 82 ms per image on a single A6000.
- With ConvNeXt-L, SED obtains 35.2% mIoU on A-150 and 22.6% on PC-459.
- Category early rejection with k=8 gives about 4.7x speedup on PC-459 with no accuracy loss (18.6 vs 18.6 mIoU).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SED with ConvNeXt-B instead of CAT-Seg with ViT-B/16 | open-vocabulary semantic segmentation · A-150 | mIoU: 27.2 -> 31.6 | Helped |
| Use SED with ConvNeXt-L instead of FC-CLIP with ConvNeXt-L | open-vocabulary semantic segmentation · PC-459 | mIoU: 18.2 -> 22.6 | Helped |
| Add the hierarchical encoder instead of the plain transformer (ablation) | SED · A-150 | mIoU: 23.7 -> 28.2 | Helped |
| Add the gradual fusion decoder (ablation) | SED with hierarchical encoder · PC-459 | mIoU: 17.2 -> 18.6 | Helped |
| Add category early rejection (ablation) | SED · A-150 | mIoU: 31.8 -> 31.6 | Hurt |
| Replace the plain encoder with a hierarchical encoder while using skip-layer fusion (ablation) | SED · A-150 | mIoU: 23.7 -> 28.2 | Helped |
| Add skip-layer fusion to the hierarchical encoder (ablation) | SED · A-847 | mIoU: 7.9 -> 9.9 | Helped |
| Fine-tune all encoder layers instead of freezing them (ablation) | SED · PAS-20 | mIoU: 77.2 -> 94.4 | Helped |
| Set the encoder learning-rate factor to 0.01 instead of 0.005 (ablation) | SED · PC-459 | mIoU: 17.6 -> 18.6 | Helped |
| Use kernel size 9 instead of 7 (ablation) | SED feature aggregation module · PC-459 | mIoU: 18.0 -> 18.6 | Helped |
| Combine spatial-level and class-level feature aggregation (ablation) | SED feature aggregation module · A-150 | mIoU: 28.8 -> 31.8 | Helped |
| Add cost-map fusion to fusion of F2, F3, and F4 (ablation) | SED skip-layer fusion module · PC-459 | mIoU: 17.9 -> 18.6 | Helped |
Surprising findings
- Stopping gradient from the skip-layer fusion module to the image encoder works better than back-propagating into it, contrary to the usual expectation that end-to-end gradients help.
Experimental setup
Datasets: COCO-Stuff
Benchmarks: ADE20K (A-847) ADE20K (A-150) PC-459 PC-59 PAS-20
Metrics: mIoU inference time (ms)
Baselines: SimBaseline OVSeg CAT-Seg SAN DeOP ZegFormer OpenSeg ODISE FC-CLIP LSeg
Builds on: CAT-Seg CLIP ConvNeXt MaskCLIP SAN
Closest papers by meaning
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.91)
- Open-Vocabulary Semantic Segmentation with Image Embedding Balancing (CVPR 2024 · similarity 0.91)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.91)
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.90)
- USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.89)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.