Research map / Detection and segmentation / Open-set and few-shot detection
Revisiting Open-Set Panoptic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds the LVIS-PS benchmark for open-set panoptic segmentation with ~1k tail categories as unknown classes and proposes a two-phase semi-supervised framework (thing-agnostic map generation plus unknown segment mining) built on a two-stage PanoFCN-2s model.
Problem
Open-set panoptic segmentation needs to detect unannotated unknown categories, but existing benchmarks use only a few (≤16) common COCO categories as unknown and annotate unknown regions as void, which is unrealistic for long-tail real-world scenes.
Approach
Construct LVIS-PS by merging LVIS tail categories into COCO panoptic maps so ~1k tail classes are unknown, train only with COCO annotations, then mine unknown segments from a class-agnostic (thing-combined) model and retrain with pseudo labels, adding semi-supervised RoI classification for unknown proposals.
Key results
- On LVIS-PS, the two-phase Semi-PanoFCN-2s framework reaches 19.80 PQ* on unseen classes versus 4.20 for Two-Phase (PanoFCN) and 0.26 for EOPSN, and it exceeds the supervised PanoFCN-2s model (17.67 PQ*).
- Cross-dataset on ADE20K, the proposed Semi-Two-Phase model gets 16.28 AP_e^m versus 14.66 for ES trained 36 epochs on COCO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine seen classes into one class rather than use class-specific labels (ablation) | PanoFCN-2s · LVIS-PS val, unseen classes | PQ*: 8.59 -> 17.67 | Helped |
| combine all thing classes into one class rather than use class-specific labels (ablation) | PanoFCN-2s · LVIS-PS val, unknown classes | PQ*-thing: 20.32 -> 29.77 | Helped |
| increase seen annotation proportion from 20% to 100% (ablation) | PanoFCN-2s with Comb-Seen · LVIS-PS val, unseen classes | PQ*: 5.78 -> 17.67 | Helped |
| increase the number of seen categories in the training annotations (ablation) | PanoFCN-2s with Comb-Seen · LVIS-PS val, unseen classes | PQ* | Helped |
| add semi-supervised training to the first phase (ablation) | Two-Phase PanoFCN-2s · LVIS-PS val, unseen classes | PQ*: 16.85 -> 19.80 | Helped |
| replace PanoFCN-2s with Panoptic FCN in the two-phase framework (ablation) | Two-Phase framework · LVIS-PS val, unseen classes | PQ*: 16.85 -> 4.20 | Hurt |
| use the two-phase framework instead of the supervised model | PanoFCN-2s · LVIS-PS val, unseen classes | PQ*: 17.67 -> 16.85 | No clear effect |
| train on LVIS-PS with COCO annotations instead of COCO | Two-Phase · ADE20K val | APᵐₑ: 14.66 -> 16.02 | Helped |
| add semi-supervised training to the first phase (ablation) | Two-Phase · ADE20K val | APᵐₑ: 16.02 -> 16.28 | Helped |
Surprising findings
- The two-phase framework's unseen-class performance matches or exceeds a supervised model that has seen-class annotations.
- Under the Comb-All setting (all thing classes merged), unknown-class segmentation performs best, suggesting class-agnostic training helps objectness learning.
Limitations
- Unknown-class pseudo labels rely on thresholds and overlap heuristics (e.g., 0.8 and 0.9 overlap thresholds) that are hand-set.
- EOPSN's clustering-based exemplar discovery performs poorly when unknown tail samples are rare and diverse.
Experimental setup
Datasets: LVIS-PS (built from LVIS and COCO) COCO ADE20K
Benchmarks: LVIS-PS val ADE20K val
Metrics: PQ SQ RQ PQ* Recall AP_e^m
Baselines: EOPSN Void-Supp Panoptic FCN ES (entity segmentation) PanoFCN-2s supervised model
Builds on: Panoptic FCN EOPSN LVIS COCO panoptic
Closest papers by meaning
- Exemplar-Based Open-Set Panoptic Segmentation Network (CVPR 2021 · similarity 0.91)
- An End-To-End Network for Panoptic Segmentation (CVPR 2019 · similarity 0.87)
- Toward Joint Thing-and-Stuff Mining for Weakly Supervised Panoptic Segmentation (CVPR 2021 · similarity 0.87)
- Fully Convolutional Networks for Panoptic Segmentation (CVPR 2021 · similarity 0.87)
- Scene-Centric Unsupervised Panoptic Segmentation (CVPR 2025 · similarity 0.86)
- Fully Data-Driven Pseudo Label Estimation for Pointly-Supervised Panoptic Segmentation (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.