Research map / Detection and segmentation / Weakly supervised detection and segmentation
Fully Data-Driven Pseudo Label Estimation for Pointly-Supervised Panoptic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a fully data-driven pseudo label branch, supervised end-to-end by point labels and assisted by a weight-shared auxiliary semantic branch, to estimate dense pseudo labels for pointly-supervised panoptic segmentation.
Problem
Pointly-supervised panoptic segmentation requires accurate dense pseudo labels from sparse point labels, but hand-crafted rules (polygon connection, traversing distance) limit pseudo label quality.
Approach
Replace hand-crafted pseudo label rules with a transformer-based pseudo label branch that uses instance-aware queries from point labels and is trained end-to-end on point labels, with an auxiliary semantic branch sharing layer weights and position embeddings to transfer semantic knowledge.
Key results
- Achieves 63.1% PQ on VOC and 40.3% PQ on COCO with ten-point labels (P10), exceeding PSPS by 6.5 and 7.2 points.
- With single-point labels (P1), achieves 52.0% PQ on VOC and 33.0% PQ on COCO, outperforming PSPS by 2.2 and 3.7 points.
- Removing the auxiliary semantic branch (lambda_sem=0) drops P1 performance from 52.0% to 32.9% PQ on VOC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace PSPS pseudo-label estimation with the fully data-driven method, using P10 labels | Pointly-supervised panoptic segmentation · MS COCO 2017 val | PQ: 33.1 -> 40.3 | Helped |
| replace PSPS pseudo-label estimation with the fully data-driven method, using P10 labels | Pointly-supervised panoptic segmentation · PASCAL VOC 2012 val | PQ: 56.6 -> 63.1 | Helped |
| replace PSPS pseudo-label estimation with the fully data-driven method, using P1 labels | Pointly-supervised panoptic segmentation · MS COCO 2017 val | PQ: 29.3 -> 33.0 | Helped |
| replace PSPS pseudo-label estimation with the fully data-driven method, using P1 labels | Pointly-supervised panoptic segmentation · PASCAL VOC 2012 val | PQ: 49.8 -> 52.0 | Helped |
| add auxiliary semantic branch supervision (λsem>0) (ablation) | Pseudo label branch · VOC val, P1 | PQ: 32.9 -> 52.0 | Helped |
| add auxiliary semantic branch supervision (λsem>0) (ablation) | Pseudo label branch · VOC val, P10 | PQ: 61.2 -> 63.1 | Helped |
| share both layer weights and position embeddings between branches (ablation) | Pseudo label branch and auxiliary semantic branch · VOC val, P1 | PQ: 49.7 -> 52.0 | Helped |
| share both layer weights and position embeddings between branches (ablation) | Pseudo label branch and auxiliary semantic branch · VOC val, P10 | PQ: 62.4 -> 63.1 | Helped |
| inject instance information into position embeddings instead of queries (ablation) | Pseudo label branch · VOC val | PQ: 52.0 -> 14.2; 63.1 -> 62.7 | Hurt |
| generate instance-aware queries from learned 2D embeddings instead of Fproj (ablation) | Query generator · VOC val | PQ: 52.0 -> 43.7; 63.1 -> 61.0 | Hurt |
| apply self-training only to the pseudo label branch (ablation) | Pseudo label branch and auxiliary semantic branch · VOC val, P1 | PQ: 50.0 -> 52.0; 48.5 -> 52.0 | Helped |
| apply self-training only to the pseudo label branch (ablation) | Pseudo label branch and auxiliary semantic branch · VOC val, P10 | PQ: 62.5 -> 63.1; 62.4 -> 63.1 | Helped |
Surprising findings
- Training the panoptic head directly with point labels yields only 22.6% PQ (P1) versus 52.0% with estimated pseudo labels, showing pseudo label estimation is critical.
- Adding self-training loss to the auxiliary semantic branch hurts performance (P1 drops from 52.0 to 50.9).
Limitations
- P1 performance on COCO (33.0%) remains well below full-mask supervision (43.6-48.0% PQ).
- Results depend on the sensitivity to lambda_sem, which differs between P1 (1) and P10 (0.1) settings.
- Point labels are simulated by random sampling from ground-truth masks rather than human annotations.
Experimental setup
Datasets: PASCAL VOC 2012 SBD MS COCO 2017
Benchmarks: PASCAL VOC 2012 val MS COCO 2017 val
Metrics: PQ PQth PQst
Baselines: PanopticFCN PanopticFCN-point PSPS JTSM Panoptic SegFormer Li et al. 2018 box-supervised Combination (Ahn and Kwak 2018)
Builds on: Panoptic SegFormer PSPS PanopticFCN color-prior loss
Closest papers by meaning
- Beyond the Label Itself: Latent Labels Enhance Semi-supervised Point Cloud Panoptic Segmentation (AAAI 2024 · similarity 0.88)
- Scene-Centric Unsupervised Panoptic Segmentation (CVPR 2025 · similarity 0.88)
- Robust Pseudo-Labeling via Decoupled Class-Aware Filtering and Dynamic Category Correction (AAAI 2026 · similarity 0.88)
- Connectivity-Driven Pseudo-Labeling Makes Stronger Cross-Domain Segmenters (NeurIPS 2024 · similarity 0.87)
- Decoupled Pseudo-labeling for Semi-Supervised Monocular 3D Object Detection (CVPR 2024 · similarity 0.86)
- W2P: Switching from Weak Supervision to Partial Supervision for Semantic Segmentation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.