Research map / Detection and segmentation / Open-set and few-shot detection
Automated Label Unification for Multi-Dataset Semantic Segmentation with GNNs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AutoUniSeg, which uses GNNs over text-embedded dataset label nodes and learnable unified label nodes to automatically learn label mappings and a unified label space for training one segmentation model on seven datasets.
Problem
Training semantic segmentation models on multiple datasets is hampered by conflicting label spaces and granularities; existing unification methods need manual relabeling/taxonomies or pairwise iterative training.
Approach
Use a GraphSAGE GNN over LLM text embeddings of dataset labels plus learnable unified label nodes; a learnable soft adjacency matrix (softmax per dataset) defines many-to-one label mappings, alternately optimized with the segmentation network via unbalanced optimal transport, so the unified label space is learned automatically in one training session.
Key results
- Mean mIoU of 56.4 across seven training datasets, versus 51.8 for Multi-SegHead and 55.9 for MSeg (with manual relabeling).
- Achieves 40.5 Meta Avg mIoU Class on WildDash 2 zero-shot (not trained on it), and 50.0 when trained on the WildDash 2 training datasets, above the prior best of 48.5 (MIX6D).
- Mean mIoU of 56.9 on unseen test datasets versus 53.2 for the best Multi-SegHead variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the GNN-constructed unified label space instead of direct label concatenation (ablation) | Multi-dataset semantic segmentation model · Seven training datasets | mIoU: 52.5 -> 56.4 | Helped |
| Use GNN label mapping instead of DBSCAN clustering on text features (ablation) | Multi-dataset semantic segmentation model · Seven training datasets | mIoU: 53.4 -> 56.4 | Helped |
| Add GNN training to learn label relationships (ablation) | Multi-dataset semantic segmentation model without GNN training · Seven training datasets | mIoU: 54.9 -> 56.4 | Helped |
| Add GPT-generated label descriptions (ablation) | Multi-dataset semantic segmentation model without GPT's label description · Seven training datasets | mIoU: 55.8 -> 56.4 | Helped |
| Train one unified model across seven datasets instead of Multi-SegHead | HRNet-W48 semantic segmentation · Seven training datasets, mean | mIoU: 51.8 -> 56.4 | Helped |
| Train one unified model across seven datasets instead of single-dataset models | HRNet-W48 semantic segmentation · Unseen datasets, mean | mIoU: 45.1 -> 56.9 | Helped |
| Train on broader, domain-general datasets rather than driving-only datasets | Semantic segmentation model · Non-driving datasets SUN, ADE, and COCO | mIoU: 4.4 vs 47.5; 5.4 vs 42.0; 7.3 vs 46.7 | Helped |
| Train a driving-domain-specific model rather than the domain-general model | Semantic segmentation model · Driving datasets CS, MPL, BDD, and IDD | mIoU: 80.7 vs 82.2; 43.7 vs 45.7; 65.5 vs 68.8; 68.6 vs 71.4 | Helped |
| Use the proposed model instead of UniSeg | WildDash 2 zero-shot evaluation · WildDash 2 | Meta Avg mIoU Class: 39.4 -> 40.5 | Helped |
| Use the proposed model instead of UniSeg | WildDash 2 zero-shot evaluation · WildDash 2 negative test cases | Negative mIoU class: 34.8 -> 43.4 | Helped |
Surprising findings
- Domain-specific driving-only model outperforms the domain-general 7-dataset model on driving datasets, but the general model wins by a wide margin on non-driving datasets.
Limitations
- Still requires fully annotated datasets; does not handle weakly or unsupervised data.
- ChatGPT-generated label descriptions may be inaccurate and affect learned label relations.
- Automatically constructed label space carries safety risks for autonomous driving and needs manual review.
Experimental setup
Datasets: CityScapes Mapillary Vistas BDD IDD SUN RGBD ADE20K COCO WildDash 2 KITTI CamVid ScanNet-20 PASCAL VOC
Benchmarks: CityScapes Mapillary Vistas SUN RGBD BDD IDD ADE20K COCO WildDash 2 KITTI CamVid ScanNet-20 PASCAL VOC
Metrics: mIoU Meta Avg mIoU Class iIoU
Baselines: MSeg NLL+ Uni NLL+ Auto univ. Single dataset Multi-SegHead EffPS SeamSeg UniSeg FAN MIX6D SNpRN152
Builds on: GraphSAGE Multi-SegHead UniSeg
Closest papers by meaning
- LMSeg: Language-guided Multi-dataset Segmentation (ICLR 2023 · similarity 0.88)
- Joining datasets via data augmentation in the label space for neural networks (ICML 2021 · similarity 0.87)
- Single-Stage Semantic Segmentation From Image Labels (CVPR 2020 · similarity 0.87)
- Dynamic-Structured Semantic Propagation Network (CVPR 2018 · similarity 0.87)
- Transferable Semi-Supervised Semantic Segmentation (AAAI 2018 · similarity 0.87)
- Label-Efficient Few-Shot Semantic Segmentation with Unsupervised Meta-Training (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.