Research map / Detection and segmentation / Open-set and few-shot detection
Task-aligned Part-aware Panoptic Segmentation through Joint Object-Part Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAPPS, which uses shared object-part queries in a Mask2Former-based network to jointly predict object segments and their compatible part segments for part-aware panoptic segmentation, achieving new state of the art on Pascal-PP and Cityscapes-PP.
Problem
Part-aware panoptic segmentation requires parts to be segmented and linked to their parent objects, but existing methods predict objects and parts separately with part-level semantic segmentation, so their learning objective is misaligned with the task.
Approach
A set of shared queries jointly predicts an object-level mask and class plus part masks only for part classes compatible with the predicted object class (JOPS head), so part predictions are instance-aware and automatically linked to parent objects.
Key results
- On Pascal-PP with COCO pre-training, TAPPS reaches PartPQ 67.2 vs 64.8 for the separate-query baseline, with PartPQ-Pt up 2.4 points.
- With Swin-B backbone on Pascal-PP, TAPPS achieves PartPQ 72.2 and PartPQ-Pt 78.1 range, and overall PartPQ 60.4 when compared to RN-50 state of the art, a gain of +6.3.
- On Cityscapes-PP with COCO pre-training, TAPPS improves PartPQ from 64.8 to... (reported 64.8 SOTA PartPQ, +1.7 over prior best).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace separate object and part queries with joint shared object-part queries | TAPPS vs. baseline · Pascal-PP val | PartPQ All: 42.9 -> 44.6; 53.7 -> 54.7 | Helped |
| Replace separate object and part queries with joint shared object-part queries | TAPPS vs. baseline · Cityscapes-PP val | PartPQ All: 58.4 -> 59.3; 60.4 -> 61.3 | Helped |
| Train and test while allowing incompatible part predictions (ablation) | TAPPS · Pascal-PP val | PartPQ All: 43.9 -> 44.6 | Hurt |
| Use one adaptation layer instead of the two-layer MLP (ablation) | TAPPS JOPS head · Pascal-PP val | PartPQ All: 54.7 -> 54.6 | No clear effect |
| Use no adaptation layers instead of the two-layer MLP (ablation) | TAPPS JOPS head · Pascal-PP val | PartPQ All: 54.7 -> 54.1 | Hurt |
| Use three adaptation layers instead of the two-layer MLP (ablation) | TAPPS JOPS head · Pascal-PP val | PartPQ All: 54.7 -> 54.6 | No clear effect |
| Generate part masks and classes dynamically rather than with fixed part-class layers (ablation) | TAPPS · Pascal-PP val | PartPQ All: 54.7 -> 54.2 | Hurt |
Surprising findings
- Thing PQ gains come mainly from better instance separation rather than better recognition, as mIoU-Th barely changes.
- Predicting only compatible parts at test time removes all object-part conflicts but yields little segmentation-quality gain unless also applied during training.
Limitations
- Part classes are defined only for a subset of thing classes, so the approach relies on dataset-specific part definitions.
- Fixed per-part-class FC layers scale with the number of part classes N_pc.
Experimental setup
Datasets: Cityscapes-Panoptic-Parts Pascal-Panoptic-Parts COCO panoptic ImageNet
Benchmarks: Cityscapes-PP Pascal-PP Pascal-PP-107
Metrics: PartPQ PartPQ-Pt PartSQ-Pt PQ PQ-Th mIoU-Th
Baselines: Panoptic-PartFormer Panoptic-PartFormer++ JPPF SegFormer + CondInst + BPR Separate-query baseline (TAPPS variant)
Builds on: Mask2Former Part-aware panoptic segmentation (De Geus et al.) Panoptic-PartFormer
Closest papers by meaning
- Part-Aware Panoptic Segmentation (CVPR 2021 · similarity 0.92)
- Panoptic Segmentation (CVPR 2019 · similarity 0.88)
- Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (CVPR 2022 · similarity 0.88)
- Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation (NeurIPS 2020 · similarity 0.88)
- Scene-Centric Unsupervised Panoptic Segmentation (CVPR 2025 · similarity 0.87)
- Fine-Grained Image-Text Correspondence with Cost Aggregation for Open-Vocabulary Part Segmentation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.