Research map / Detection and segmentation / Open-set and few-shot detection
Learning Content-Enhanced Mask Transformer for Domain Generalized Urban-Scene Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CMFormer, a Mask2Former-based Transformer with content-enhanced mask attention that fuses high-resolution mask features with down-sampled counterparts to improve domain-generalized urban-scene segmentation.
Problem
Domain-generalized urban-scene semantic segmentation, where semantic content is similar across domains but styles (weather, lighting, landscape) vary strongly; prior methods are CNN-based style de-coupling or adversarial training.
Approach
A content-enhanced mask attention (CMA) that learns mask queries from both the full-resolution image feature (good content representation) and its average-pooled down-sampled version (robust to style), fused by concatenation and a linear layer, applied across multiple resolutions in a Mask2Former-style decoder.
Key results
- Trained on GTA5, CMFormer reaches 55.31/49.91/60.09/43.80 mIoU on C/B/M/S, gains of up to 14.00% over prior CNN-based methods.
- Trained on Cityscapes with Swin-L, CMFormer reaches 62.6/73.6/60.7/43.0 on B/M/G/S, improving on HGFormer by 1.1-1.7 mIoU.
- Ablation on Cityscapes-trained Swin-B: adding content enhancement at x32, x16, x8 raises mean target mIoU from 55.43 to 59.27 on B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the strongest CNN baseline with CMFormer | GTA5-source domain generalization · GTA5→Cityscapes, BDD-100K, Mapillary, SYNTHIA | mIoU: +10.66, +9.45, +14.00, and +12.46% | Helped |
| Replace the strongest CNN baseline with CMFormer | SYNTHIA-source domain generalization · SYNTHIA→Cityscapes, Mapillary, GTA5 | mIoU: 5.67%, 8.73%, and 11.49% | Helped |
| Replace SAW with CMFormer | SYNTHIA-source domain generalization · SYNTHIA→BDD-100K | mIoU: 1.80% | Hurt |
| Use CMFormer instead of HGFormer | Cityscapes-source domain generalization with Swin-L backbone · Cityscapes→BDD-100K, Mapillary, GTA5, SYNTHIA | mIoU: 1.1%, 1.5%, 1.3%, and 1.7% | Helped |
| Use CMFormer instead of existing methods | Synthetic GTA5+SYNTHIA source domain · synthetic-to-real: Cityscapes, BDD-100K, Mapillary | mIoU: >10% | Helped |
| Use CMFormer instead of existing methods | Cityscapes-trained model evaluated on ACDC · fog, night, rain, snow | mIoU: 10.3%, 0.5%, 11.6%, 11.1% | Helped |
| Add content enhancement at ×32 (ablation) | CMFormer · Cityscapes→BDD-100K, Mapillary, GTA5, SYNTHIA | mIoU: 0.74%, 1.43%, 0.37%, 0.64% | Helped |
| Add content enhancement at ×16 after ×32 (ablation) | CMFormer · Cityscapes→BDD-100K, Mapillary, GTA5, SYNTHIA | mIoU: 1.93%, 2.17%, 0.12%, 0.58% | Helped |
| Add content enhancement at ×8 after ×32 and ×16 (ablation) | CMFormer · Cityscapes→BDD-100K, Mapillary, GTA5, SYNTHIA | mIoU: 55.54 -> 58.11; 39.41 -> 40.43 | Helped |
Surprising findings
- On SYNTHIA-source training, CMFormer underperforms SAW on the BDD100K target by 1.80 mIoU.
- Baseline numbers are cited from prior work under ResNet-50 while CMFormer uses Swin backbones, so comparisons are not backbone-matched.
Limitations
- Performance gain is smaller on SYNTHIA-trained setting, attributed to fewer training samples and transformer under-training.
- Baselines are reported from prior works with ResNet-50 backbones rather than re-run with matched backbones.
- Comparison with HGFormer is limited to one-decimal reported numbers.
Experimental setup
Datasets: Cityscapes BDD100K Mapillary SYNTHIA GTA5 ACDC ImageNet
Benchmarks: Cityscapes BDD100K Mapillary SYNTHIA GTA5 ACDC
Metrics: mIoU
Baselines: IBN IW Iternorm DRPC ISW GTR DIRL SHADE SAW WildNet AdvStyle SPC
Builds on: Mask2Former MaskFormer Swin Transformer ISW SAW
Closest papers by meaning
- Learning Generalized Segmentation for Foggy-Scenes by Bi-directional Wavelet Guidance (AAAI 2024 · similarity 0.88)
- MaskViM: Domain Generalized Semantic Segmentation with State Space Models (AAAI 2025 · similarity 0.87)
- Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.87)
- Rethinking Decoders for Transformer-based Semantic Segmentation: A Compression Perspective (NeurIPS 2024 · similarity 0.87)
- Learning Frequency-Adapted Vision Foundation Model for Domain Generalized Semantic Segmentation (NeurIPS 2024 · similarity 0.87)
- Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.