Research map / Detection and segmentation / Open-set and few-shot detection
OMG-Seg: Is One Model Good Enough For All Segmentation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OMG-Seg, a single Mask2Former-style encoder-decoder with a frozen CLIP ConvNeXt backbone and a shared query-based decoder, co-trained jointly to handle over ten image, video, open-vocabulary and SAM-style interactive segmentation tasks.
Problem
Existing segmentation models are task-specific or only partially unified; few handle image, video, open-vocabulary, and promptable interactive segmentation within one model without large performance loss.
Approach
Unify all segmentation outputs as object queries (mask, label, ID, or visual prompt) processed by one shared Mask2Former-style decoder, with a frozen CLIP visual backbone and CLIP text embeddings as classifiers, and jointly co-train on image, video, and prompt datasets.
Key results
- OMG-Seg (ConvNeXt-Large, frozen) reaches 53.8 COCO-PS PQ, 49.8 VIPSeg VPQ and 56.4 YT-VIS-19 mAP in a single shared model.
- In multi-dataset co-training, OMG-Seg with 221M parameters matches or beats single-dataset baselines totaling 1326M parameters on VIPSeg-VPS (46.9 vs 42.3) and YT-VIS-19 (48.8 vs 45.3).
- Scaling to ConvNeXt-XX-Large (frozen) gives 55.4 COCO-PS PQ and 53.1 VIPSeg VPQ, with 59.3 COCO-SAM mIoU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VIPSeg and YT-VIS-19 training data to COCO co-training (ablation) | OMG-Seg · VIPSeg-VPS | VPQ: 32.2 -> 49.0 | Helped |
| Add VIPSeg and YT-VIS-19 training data to COCO co-training (ablation) | OMG-Seg · COCO-PS | PQ: 53.4 -> 53.0 | No clear effect |
| Add YT-VIS-19 training data to COCO and VIPSeg co-training (ablation) | OMG-Seg · YT-VIS-19 | mAP: 45.2 -> 56.8 | Helped |
| Use a shared decoder instead of decoupled image/video decoders (ablation) | OMG-Seg · VIPSeg-VPS | VPQ: 46.2 -> 48.5 | Helped |
| Use a shared decoder instead of decoupled image/video decoders (ablation) | OMG-Seg · COCO-PS | PQ: 53.6 -> 53.0 | Hurt |
| Add an adapter to the frozen CLIP backbone (ablation) | OMG-Seg · COCO-PS | PQ: 53.0 -> 53.5; 54.8 -> 54.6 | Mixed |
| Add an adapter to the frozen CLIP backbone (ablation) | OMG-Seg · VIPSeg-VPS | VPQ: 48.5 -> 49.2; 50.1 -> 49.6 | Mixed |
| Increase CLIP backbone size from ResNet50 to ConvNeXt-Large (ablation) | OMG-Seg · COCO-PS | PQ: 44.8 -> 53.0 | Helped |
| Increase CLIP backbone size from ResNet50 to ConvNeXt-Large (ablation) | OMG-Seg · VIPSeg-VPS | VPQ: 42.0 -> 48.5 | Helped |
| Increase CLIP backbone size from ResNet50 to ConvNeXt-Large (ablation) | OMG-Seg · ADE-OV | PQ: 18.2 -> 26.8 | Helped |
Surprising findings
- Performance on ADE-20k drops sharply under joint co-training (e.g., 28.2 PQ vs 41.2 for single-dataset baseline).
- Using a pre-trained (trained) backbone improves image segmentation but slightly hurts video segmentation, described as a tug-of-war over backbone capacity.
Limitations
- Gap to specialized models on COCO panoptic and instance segmentation, attributed to short 24-epoch training and frozen backbone.
- Joint co-training with video and interactive data is harder than prior work, with less optimal results on some image benchmarks.
- Excludes medical and referring segmentation tasks from scope.
Experimental setup
Datasets: COCO panoptic COCO-SAM VIPSeg YouTube-VIS-2019 ADE-20k YouTube-VIS-2021 DAVIS-2017 Cityscapes
Benchmarks: COCO panoptic Cityscapes-PS COCO instance VIPSeg-VPS YouTube-VIS-2019 YouTube-VIS-2021-OV ADE-20k open vocabulary DAVIS-2017 VOS COCO-SAM
Metrics: PQ mAP VPQ J&F mIoU
Baselines: Mask2Former Tube-Link TarViS OneFormer ODISE FC-CLIP UNINEXT HIPIE Semantic-SAM SAM K-Net
Builds on: Mask2Former SAM OpenCLIP MMDetection
Closest papers by meaning
- Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search (ICML 2026 · similarity 0.90)
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.90)
- EOV-Seg: Efficient Open-Vocabulary Panoptic Segmentation (AAAI 2025 · similarity 0.88)
- SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.88)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.88)
- USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.