Research map / Detection and segmentation / Open-set and few-shot detection
Unleashing the Power of Visual Foundation Models for Generalizable Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts visual foundation models with LoRA and a lightweight deconvolution decoder (VFMNet), then adds a mask-guided high-resolution refinement network (MGRNet) and coarse-to-fine sliding-window inference for domain generalized semantic segmentation.
Problem
Visual foundation models lose generalization when heavy decoders are attached and perform poorly on high-resolution inputs due to length extrapolation, hurting domain generalized semantic segmentation (e.g., small objects in unseen driving domains).
Approach
Fine-tune a VFM encoder minimally with LoRA and a simple deconvolution decoder to preserve generalizable knowledge, then refine low-confidence high-resolution patches with a mask-guided network that uses the low-resolution coarse prediction as a class prior via feature-to-semantics attention and random feature-token masking.
Key results
- Ours with DINOv2-L reaches 70.10% average mIoU on GTAV→Citys+BDD+Map, versus 66.67% for reproduced Rein†, and is reported as a 3.4% gain over the state of the art.
- Under Citys→BDD+Map generalization, the DINOv2-L version reaches 71.62% average mIoU, a 1.1% improvement over the prior best.
- VFMNet alone raises the frozen DINOv2 baseline from 58.73% to 69.00% average mIoU; MGRNet and feature masking add 0.35 and 0.75 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the VFM with VFMNet rather than using a frozen DINOv2 backbone with a linear decoder (ablation) | DINOv2-L semantic segmentation · GTAV → Avg. | mIoU: 58.73 -> 69.00 | Helped |
| Add MGRNet to VFMNet (ablation) | DINOv2-L semantic segmentation · GTAV → Avg. | mIoU: 69.00 -> 69.35 | Helped |
| Add feature masking (ablation) | DINOv2-L semantic segmentation with MGRNet · GTAV → Avg. | mIoU: 69.35 -> 70.10 | Helped |
| Use the proposed deconvolution decoder instead of a linear decoder (ablation) | VFMNet with DINOv2-L · GTAV → Avg. | mIoU: 67.94 -> 69.00 | Helped |
| Use a SegFormer decoder instead of the proposed decoder (ablation) | VFMNet with DINOv2-L · GTAV → Avg. | mIoU: 69.00 -> 68.04 | Hurt |
| Use a Mask2Former decoder instead of the proposed decoder (ablation) | VFMNet with DINOv2-L · GTAV → Avg. | mIoU: 69.00 -> 67.64 | Hurt |
| Set feature mask ratio p to 0.1 instead of 0.2 (ablation) | MGRNet · GTAV → Avg. | mIoU: 70.10 -> 69.56 | Hurt |
| Set feature mask ratio p to 0.3 instead of 0.2 (ablation) | MGRNet · GTAV → Avg. | mIoU: 70.10 -> 69.87 | Hurt |
| Set feature mask ratio p to 0.4 instead of 0.2 (ablation) | MGRNet · GTAV → Avg. | mIoU: 70.10 -> 69.33 | Hurt |
| Set feature mask ratio p to 0 instead of 0.2 (ablation) | MGRNet · GTAV → Avg. | mIoU: 70.10 -> 69.35 | Hurt |
| Set detail-loss weight λ to 1.0 (ablation) | VFMNet and MGRNet · GTAV → Avg. | mIoU: 69.27; 69.61; 70.10; 69.63; 69.48 | Helped |
Surprising findings
- More complex decoders (Mask2Former) did not improve over a simple linear head after LoRA fine-tuning (67.64% vs 67.94%).
- CLIP and SAM backbones performed relatively poorly compared with DINOv2 and EVA02 in this setup.
Limitations
- Evaluated only on driving-scene segmentation datasets (GTAV, Cityscapes, BDD100K, Mapillary).
- Requires two networks and a multi-stage sliding-window inference, adding inference complexity for high-resolution images.
- Performance drop of CLIP and SAM backbones is attributed to speculation rather than verified.
Experimental setup
Datasets: GTAV Cityscapes
Benchmarks: Cityscapes BDD100K Mapillary Vistas
Metrics: mIoU
Baselines: SAN-SAW WildNet SHADE TLDR FAMix HGFormer CMFormer VLTSeg Rein
Builds on: Rein LoRA Mask2Former MMSegmentation
Closest papers by meaning
- Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.90)
- Leveraging Depth and Language for Open-Vocabulary Domain-Generalized Semantic Segmentation (NeurIPS 2025 · similarity 0.88)
- MRFP: Learning Generalizable Semantic Segmentation from Sim-2-Real with Multi-Resolution Feature Perturbation (CVPR 2024 · similarity 0.88)
- FisherTune: Fisher-Guided Robust Tuning of Vision Foundation Models for Domain Generalized Segmentation (CVPR 2025 · similarity 0.88)
- MaskViM: Domain Generalized Semantic Segmentation with State Space Models (AAAI 2025 · similarity 0.87)
- Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.