Research map / Detection and segmentation / Open-set and few-shot detection
Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Assesses vision foundation models for domain generalized semantic segmentation and proposes Rein, a parameter-efficient fine-tuning method using instance-linked learnable tokens inserted between frozen ViT layers.
Problem
Domain generalized semantic segmentation (DGSS) trained on small urban-scene datasets, where full fine-tuning of large vision foundation models overfits and limits generalization to unseen domains.
Approach
Rein inserts layer-wise refinement between frozen VFM layers, using learnable low-rank tokens linked to instances via dot-product similarity maps (with query-based linkage to Mask2Former), and shares MLP weights across layers so only about 1% of parameters are trained while surpassing full fine-tuning.
Key results
- Rein with DINOv2-L (2.99M trainable params) reaches 64.3% average mIoU across Cityscapes, BDD100K and Mapillary under GTAV training, versus 61.1 for frozen DINOv2 and 61.7 for full fine-tuning.
- Rein with EVA02-L reaches 63.6% average mIoU, outperforming full fine-tuning (60.9) and AdaptFormer (62.6) under the same setting.
- Adding UrbanSyn to GTAV and Synthia, Rein reaches 78.4% mIoU on Cityscapes validation without real urban-scene data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Rein instead of freezing the backbone | DINOv2-L · GTAV → Cityscapes + BDD100K + Mapillary | average mIoU: 61.1 -> 64.3 | Helped |
| Apply Rein instead of full parameter fine-tuning | DINOv2-L · GTAV → Cityscapes + BDD100K + Mapillary | average mIoU: 61.7 -> 64.3 | Helped |
| Add synthetic Synthia training data to GTAV | Rein · Cityscapes val | mIoU: 66.4 -> 68.1 | Helped |
| Add synthetic UrbanSyn training data | Rein · Cityscapes val | mIoU: 68.1 -> 78.4 | Helped |
| Add Rein core (ablation) | EVA02-L · GTAV → Cityscapes | mIoU: Freeze 56.5 -> Rein-core 63.4 | Helped |
| Add Rein-link (ablation) | EVA02-L with Rein core · GTAV → Cityscapes | mIoU: 63.4 -> 64.1 | Helped |
| Use layer-shared MLP weights (ablation) | EVA02-L with Rein-link · GTAV → Cityscapes | mIoU: 64.1 -> 63.4 | Hurt |
| Use low-rank token sequence (ablation) | EVA02-L with shared MLP weights · GTAV → Cityscapes | mIoU: 63.4 -> 65.3 | Helped |
| Add Rein core (ablation) | DINOv2-L · GTAV → Cityscapes | mIoU: Freeze 61.1 -> Rein-core 64.9 | Helped |
| Add Rein-link (ablation) | DINOv2-L with Rein core · GTAV → Cityscapes | mIoU: 64.9 -> 65.8 | Helped |
| Use layer-shared MLP weights (ablation) | DINOv2-L with Rein-link · GTAV → Cityscapes | mIoU: 65.8 -> 65.8 | No clear effect |
| Use low-rank token sequence (ablation) | DINOv2-L with shared MLP weights · GTAV → Cityscapes | mIoU: 65.8 -> 66.4 | Helped |
Surprising findings
- Frozen VFMs with a simple decode head already outperform previous DGSS methods designed for CNN backbones, without specialized design.
- Classic DGSS augmentation methods (AdvStyle, PASTA, GTR) give little or negative gains when applied to VFMs.
Limitations
- Gains on some categories are lower than frozen or full baselines (e.g., 'road, sidewalk' in some settings), suggesting incomplete knowledge retention.
- Evaluation is limited to urban-scene driving datasets and Mask2Former decode head in main experiments.
Experimental setup
Datasets: GTAV Cityscapes BDD100K Mapillary Synthia UrbanSyn ACDC Cityscapes-C
Benchmarks: Cityscapes BDD100K Mapillary ACDC Cityscapes-C
Metrics: mIoU
Baselines: Full fine-tuning Freeze AdvStyle PASTA GTR-LTR LoRA AdaptFormer VPT HGFormer WildNet
Builds on: Mask2Former LoRA VPT AdaptFormer
Closest papers by meaning
- FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentation (CVPR 2025 · similarity 0.91)
- Unleashing the Power of Visual Foundation Models for Generalizable Semantic Segmentation (AAAI 2025 · similarity 0.90)
- Collaborating Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.90)
- Leveraging Depth and Language for Open-Vocabulary Domain-Generalized Semantic Segmentation (NeurIPS 2025 · similarity 0.90)
- Learning Frequency-Adapted Vision Foundation Model for Domain Generalized Semantic Segmentation (NeurIPS 2024 · similarity 0.89)
- MRFP: Learning Generalizable Semantic Segmentation from Sim-2-Real with Multi-Resolution Feature Perturbation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.