Research map / Detection and segmentation / Open-set and few-shot detection
JRDB-PanoTrack: An Open-world Panoptic Segmentation and Tracking Robotic Dataset in Crowded Human Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces JRDB-PanoTrack, a multi-modal robot dataset with 2D panoptic segmentation and tracking annotations (multi-label, closed- and open-world splits) plus OSPA-based metrics, and benchmarks SOTA segmenters and trackers on it.
Problem
Existing panoptic segmentation and tracking datasets are mostly from self-driving or internet sources, with limited classes, single modalities, and no multi-label or open-world robotic coverage, so they do not capture crowded human-centric robot environments.
Approach
Builds panoptic segmentation and tracking annotations on the JRDB robot dataset with multi-label masks for objects seen behind glass or hung on walls, splits 72 classes into known and unknown sets for open-world evaluation, and proposes OSPA-based metrics (OSPA_PS and OSPA2_PT) that handle multi-label and ID recovery.
Key results
- Mask DINO achieves the best closed-world panoptic segmentation PQ of 36.57% on JRDB-PanoTrack with COCO pretraining, versus 31.41% without.
- Open-world segmentation drops sharply: FC-CLIP reaches PQ 10.06 on JRDB-PanoTrack (vs. 26.8 on ADE20K), with ODISE-C at 11.07 overall.
- Training Mask DINO on JRDB-PanoTrack then finetuning on COCO raises COCO PQ from 42.16 to 44.48.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COCO pretraining before fine-tuning on JRDB-PanoTrack (ablation) | Mask DINO · JRDB-PanoTrack test, CW panoptic segmentation | PQ: 31.41 -> 36.57 | Helped |
| Train FC-CLIP in-domain on JRDB-PanoTrack known classes rather than cross-domain on COCO (ablation) | FC-CLIP · JRDB-PanoTrack test, OW known classes | PQ: 14.29 -> 24.95 | Helped |
| Train FC-CLIP in-domain on JRDB-PanoTrack known classes rather than cross-domain on COCO (ablation) | FC-CLIP · JRDB-PanoTrack test, OW unknown classes | PQ: 3.56 -> 3.19 | Hurt |
| Pretrain Mask DINO on JRDB-PanoTrack before fine-tuning on COCO (ablation) | Mask DINO · COCO val | PQ: 42.16 -> 44.48 | Helped |
| Use BoT-SORT rather than OC-SORT for track association | CW panoptic tracking methods on JRDB-PanoTrack · JRDB-PanoTrack test, CW panoptic tracking | STQ: 7.40 -> 14.20; 7.09 -> 13.89; 7.70 -> 14.50 | Helped |
| Use BoT-SORT rather than OC-SORT for track association | OW panoptic tracking methods on JRDB-PanoTrack · JRDB-PanoTrack test, OW panoptic tracking | STQ: 2.50 -> 4.71; 3.11 -> 6.49; 4.19 -> 8.32; 4.90 -> 9.01 | Helped |
| Use MaskPLS rather than DSNet for 3D panoptic segmentation | 3D panoptic segmentation · JRDB-PanoTrack test | PQ: 3.41 -> 7.02 | Helped |
| Use Mask4D rather than DSNet for 3D panoptic tracking | 3D panoptic tracking · JRDB-PanoTrack test | LSTQ: 25.35 -> 27.87 | Helped |
Surprising findings
- Pretraining on JRDB-PanoTrack and fine-tuning on COCO improves COCO results, indicating the dataset transfers to other domains.
- In open-world tracking, BoT-SORT gives higher STQ but more fragmentation, while OC-SORT yields lower OSPA errors and fewer fragmentations.
Limitations
- Multi-label areas had to be preprocessed into single-label for standard metrics because no pretrained multi-label models exist
- 3D pseudo-labels projected from 2D annotations may contain noise
- Open-world performance on unknown classes remains very low
Experimental setup
Datasets: JRDB-PanoTrack COCO
Benchmarks: JRDB-PanoTrack COCO
Metrics: PQ OSPA_PS STQ IDF1 OSPA2_PT Frag IoU LSTQ
Baselines: Kmax-Deeplab Mask2Former ODISE-L ODISE-C FC-CLIP ByteTrack OC-SORT BoT-SORT DSNet Mask4D MaskPLS
Builds on: JRDB Mask DINO ODISE FC-CLIP BoT-SORT
Closest papers by meaning
- JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups (CVPR 2024 · similarity 0.85)
- MCD: Diverse Large-Scale Multi-Campus Dataset for Robot Perception (CVPR 2024 · similarity 0.85)
- RoboSense: Large-scale Dataset and Benchmark for Egocentric Robot Perception and Navigation in Crowded and Unstructured Environments (CVPR 2025 · similarity 0.84)
- PanoWorld: Towards Spatial Supersensing in 360◦ Panorama World (NeurIPS 2026 · similarity 0.84)
- RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics (CVPR 2025 · similarity 0.84)
- FreeMan: Towards Benchmarking 3D Human Pose Estimation under Real-World Conditions (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.