Research map / Detection and segmentation / Open-set and few-shot detection
Scene-Centric Unsupervised Panoptic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CUPS, an unsupervised panoptic segmentation method that trains directly on scene-centric images using pseudo labels from motion (SF2SE3 on scene flow), depth-guided DINO-based semantics, and self-training.
Problem
Unsupervised panoptic segmentation has relied on object-centric training data (e.g., ImageNet/COCO), which fails on complex scene-centric imagery; the task needs annotation-free panoptic segmentation of scene-centric images.
Approach
Generates high-resolution panoptic pseudo labels on scene-centric stereo video by fusing motion-based instance masks (ensembled SF2SE3 on scene flow) with depth-guided semantic inference over distilled DINO features, then bootstraps a panoptic network with DropLoss and self-enhanced copy-paste and refines it via EMA momentum-network self-training.
Key results
- On Cityscapes val, CUPS reaches 27.8% PQ versus 18.4% for U2Seg (+9.4 points), the previous state of the art.
- CUPS improves unsupervised semantic segmentation on Cityscapes to 26.8% mIoU versus 23.1% for DepthG.
- Using only 60 annotated Cityscapes images (2%), label-efficient fine-tuning reaches 43.6% PQ, about 70% of the supervised upper bound.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CUPS instead of U2Seg | Unsupervised panoptic segmentation · Cityscapes val | PQ: 18.4 -> 27.8 | Helped |
| Use CUPS instead of U2Seg | Unsupervised panoptic segmentation · KITTI, BDD, MUSES, Waymo, and MOTS (OOD) | PQ: +4.9; +4.1; +4.1; +6.6; +17.1 | Helped |
| Use CUPS instead of DepthG | Unsupervised semantic segmentation · Cityscapes val | mIoU: 23.1 -> 26.8 | Helped |
| Use CUPS instead of MOD-UV | Class-agnostic instance segmentation · Waymo | AP50: 25.1 -> 30.5 | Helped |
| Add instance-aligned semantics to vanilla semantics + SF2SE3 (ablation) | CUPS pseudo-label generation · Cityscapes val pseudo labels | PQ: 14.3 -> 14.9 | Helped |
| Add SF2SE3 ensembling (ablation) | CUPS pseudo-label generation · Cityscapes val pseudo labels | PQ: 14.9 -> 15.9 | Helped |
| Add depth-guided semantic inference (ablation) | CUPS pseudo-label generation · Cityscapes val pseudo labels | PQ: 15.9 -> 18.1 | Helped |
| Vary the thing-stuff threshold ψts (ablation) | CUPS pseudo-label generation · Cityscapes val pseudo labels | PQ: 17.8; 18.1; 18.1; 18.1; 17.7; 16.7 | No clear effect |
| Use CUPS pseudo labels instead of vanilla pseudo labels (ablation) | CUPS panoptic training · Cityscapes val | PQ: 19.7 -> 27.8 | Helped |
| Increase the number of pseudo classes from 27 to 40 or 54 (ablation) | CUPS panoptic training · Cityscapes val | PQ: 27.8 -> 30.3 -> 30.6 | Helped |
| Add DropLoss (ablation) | CUPS panoptic training · Cityscapes val | PQ: 24.1 -> 25.3 | Helped |
| Add copy-paste augmentation (ablation) | CUPS panoptic training · Cityscapes val | PQ: 25.3 -> 26.3 | Helped |
Surprising findings
- Increasing the number of pseudo classes from 27 to 54 raises PQ further (27.8 to 30.6 on Cityscapes-trained model), indicating over-clustering helps.
- MaskCut attends to semantically correlated areas rather than instances on scene-centric images, with mask precision of 6.5% versus 59.6% for CUPS instance labels.
Limitations
- Relies on stereo video for scene flow and depth, so it requires stereo sequences for pseudo-label generation.
- Lower PQ on small and rare categories; CUPS is slightly worse than MOD-UV on small instances in Waymo.
- Output semantic classes still require mapping to ground-truth taxonomy via matching; unsupervised labels are not aligned with human categories.
Experimental setup
Datasets: Cityscapes KITTI Waymo BDD MUSES MOTS ImageNet
Benchmarks: Cityscapes val KITTI panoptic BDD MUSES Waymo MOTS
Metrics: PQ SQ RQ mIoU pixel accuracy mask AP AP50
Baselines: U2Seg DepthG + CutLER STEGO DepthG MOD-UV CutLER HASSOD PiCIE DiffSeg
Builds on: CutLER U2Seg SF2SE3 DepthG SMURF STEGO
Closest papers by meaning
- Unifying Training and Inference for Panoptic Segmentation (CVPR 2020 · similarity 0.92)
- Scene-Centric Unsupervised Video Panoptic Segmentation (CVPR 2026 · similarity 0.91)
- Unsupervised Universal Image Segmentation (CVPR 2024 · similarity 0.90)
- Fully Data-Driven Pseudo Label Estimation for Pointly-Supervised Panoptic Segmentation (AAAI 2024 · similarity 0.88)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.88)
- Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.