Research map / Label-efficient and robust learning / Domain adaptation and generalization
MRFP: Learning Generalizable Semantic Segmentation from Sim-2-Real with Multi-Resolution Feature Perturbation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MRFP, a training-only feature perturbation module (overcomplete random autoencoder for high-resolution fine features plus normalized perturbation for style) that improves Sim-2-Real single- and multi-domain generalization for urban semantic segmentation with DeepLabv3+.
Problem
Semantic segmentation models trained on synthetic urban-scene data (GTAV, Synthia) generalize poorly to unseen real-world target domains due to lack of style diversity and domain-specific fine-grained features.
Approach
Perturb low-frequency style statistics via normalized perturbation (NP+) and high-frequency domain-specific fine features via a randomly initialized overcomplete autoencoder (HRFP) whose decreasing receptive field focuses on fine details, adding no learnable parameters or losses and removed at inference.
Key results
- MRFP+ with ResNet-50 trained on GTAV reaches 39.27 average mIoU across B, C, M, S versus 31.21 for the baseline (about 7.56 points gain) and 36.63 for ProRandConv.
- On (G+S) to B, C, M, MRFP+ achieves 44.24 average mIoU versus 34.88 baseline and 37.40 for ISW.
- On adverse weather (GTAV to Foggy and Rainy Cityscapes), MRFP+ averages 25.87 mIoU versus 21.67 for WildNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add HRFP+ and NP+ to the baseline | DeepLabv3+ with ResNet-50 · BDD-100k, Cityscapes, Mapillary, and Synthia val (G→target) | mIoU: 31.44 -> 39.55 (BDD-100k); 34.66 -> 42.40 (Cityscapes); 32.93 -> 44.93 (Mapillary); 25.84 -> 30.22 (Synthia) | Helped |
| Add HRFP+ and NP+ to the baseline | DeepLabv3+ with ResNet-50 · Foggy and Rainy Cityscapes val (G→target) | mIoU: 40.67; 30.42; 23.74; 19.25; 15.28 | Helped |
| Add HRFP and NP+ to the baseline | DeepLabv3+ with MobileNetv2 · BDD-100k, Cityscapes, Synthia, Mapillary, and Foggy Cityscapes val (G→target) | mIoU: 33.03; 32.92; 25.62; 30.95; 32.97 | Helped |
| Add HRFP+ and NP+ to the baseline | DeepLabv3+ with ResNet-50 · BDD-100k, Cityscapes, and Mapillary val ((G+S)→target) | mIoU: 30.11 -> 41.13; 38.63 -> 46.18; 35.90 -> 45.28 | Helped |
| Add HRFP+ and NP+ rather than use HRFP+ alone (ablation) | DeepLabv3+ with ResNet-50 · BDD-100k, Cityscapes, Mapillary, and Synthia val (G→target) | mIoU: 39.28 -> 39.55; 41.39 -> 42.40; 42.70 -> 44.93; 29.70 -> 30.22 | Helped |
| Add HRFP+ and NP+ rather than use HRFP alone (ablation) | DeepLabv3+ with ResNet-50 · BDD-100k, Cityscapes, Mapillary, and Synthia val (G→target) | mIoU: 34.18 -> 39.55; 38.15 -> 42.40; 43.33 -> 44.93; 26.71 -> 30.22 | Helped |
| Add HRFP+ and NP+ rather than use NP+ alone (ablation) | DeepLabv3+ with ResNet-50 · BDD-100k, Cityscapes, Mapillary, and Synthia val (G→target) | mIoU: 34.50 -> 39.55; 40.33 -> 42.40; 38.85 -> 44.93; 28.65 -> 30.22 | Helped |
| Use overcomplete HRFP rather than spatially consistent feature perturbation (SCFP) (ablation) | DeepLabv3+ with ResNet-50 · BDD-100k, Cityscapes, Mapillary, and Synthia val (G→target) | mIoU: 38.57 -> 39.55; 40.65 -> 42.40; 42.48 -> 44.93; 28.24 -> 30.22 | Mixed |
| Replace HRFP+ with HRFP while keeping NP+ (ablation) | DeepLabv3+ with ResNet-50 · BDD-100k, Cityscapes, Mapillary, and Synthia val (G→target) | mIoU: 39.55 vs 38.80; 42.40 vs 40.25; 44.93 vs 41.96; 30.22 vs 27.37 | Mixed |
Surprising findings
- Perturbing high-frequency fine-grained features alone (HRFP+) outperforms the NP+ style-perturbation alone by 2.68 points, suggesting fine-feature perturbation matters as much as style perturbation.
Limitations
- Gains are reported relative to re-implemented baselines and results averaged over only three runs.
- Evaluated only on urban-scene segmentation with synthetic-to-real shift; generality to other domains is untested in the text.
Experimental setup
Datasets: GTAV Synthia Cityscapes Foggy Cityscapes BDD-100k Mapillary Rainy Cityscapes
Benchmarks: Cityscapes BDD-100k Mapillary Synthia Foggy Cityscapes Rainy Cityscapes
Metrics: mIoU
Baselines: IBN-Net ISW SAN-SAW WildNet WEDGE DIRL ProRandConv SCFP
Builds on: NP+ (normalized perturbation) DeepLabv3+ RobustNet (ISW) ProRandConv RandConv
Closest papers by meaning
- Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.89)
- Unleashing the Power of Visual Foundation Models for Generalizable Semantic Segmentation (AAAI 2025 · similarity 0.88)
- Towards the Uncharted: Density-Descending Feature Perturbation for Semi-supervised Semantic Segmentation (CVPR 2024 · similarity 0.87)
- Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation (CVPR 2025 · similarity 0.87)
- Learning Frequency-Adapted Vision Foundation Model for Domain Generalized Semantic Segmentation (NeurIPS 2024 · similarity 0.86)
- Learning Content-Enhanced Mask Transformer for Domain Generalized Urban-Scene Segmentation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.