Research map / Detection and segmentation / Open-set and few-shot detection
Leveraging Depth and Language for Open-Vocabulary Domain-Generalized Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vireo, a single-stage framework for open-vocabulary domain-generalized semantic segmentation that combines frozen VFM and Depth Anything V2 encoders with GeoText Query, Coarse Mask Prior Embedding, and a DOV-VEH decoder.
Problem
Open-vocabulary semantic segmentation for unseen classes while staying robust to unseen domains (e.g., adverse weather) in a unified model, where text-vision alignment degrades under domain shift and domain-invariant methods suppress fine-grained semantic cues.
Approach
Injects depth-derived geometric features from a frozen Depth VFM into a frozen VFM encoder via layer-wise learnable GeoText queries that align geometry with CLIP text embeddings, adds a coarse mask prior to densify gradients into the encoder, and fuses structural and semantic features in a transformer decoder head.
Key results
- Trained on GTA5 with EVA02-L, Vireo reaches 66.0% average mIoU over Cityscapes/BDD100K/Mapillary, vs 64.9% for FADA, with about 3.78M trainable parameters.
- With DINOv2-L trained on GTA5, Vireo reaches 67.7% average mIoU (+1.6 over FADA).
- On Cityscapes to ACDC, Vireo reports 60.6 Night, 82.3 Fog, 76.3 Rain, 76.2 Snow mIoU, exceeding FADA and REIN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Depth Anything V2 (ablation) | Vireo · Six evaluation scenarios | mIoU: approximate $0.6\%$ mIoU gain | Helped |
| Add depth augmentation with attention optimization (DA + AO) (ablation) | Vireo with Depth Anything V2 · Six evaluation scenarios | mIoU: further $\approx1\%$ improvement | Helped |
| Add DOV-VEH (ablation) | Vireo · Six evaluation scenarios | mIoU: $0.5\%$--$0.8\%$ | Helped |
| Add CMPE (ablation) | Vireo · Six evaluation scenarios | mIoU: $0.5\%$--$0.8\%$ | Helped |
| Add GeoText Query (ablation) | Vireo · Six evaluation scenarios | mIoU: $\approx4.4\%$ boost | Helped |
Limitations
- Assumes a reliable RGB camera; stream loss from occlusion, glare, or hardware failure impairs perception.
- Relies on a single RGB modality rather than multi-sensor fusion.
Experimental setup
Datasets: Cityscapes GTA5 BDD100K Mapillary ACDC ADE20K (ADE150, ADE847) DELIVER
Benchmarks: Cityscapes ACDC BDD100K Mapillary GTA5 DELIVER ADE150 ADE847
Metrics: mIoU
Baselines: FC-CLIP EBSeg CAT-Seg SED IBN RobustNet WildNet HGFormer CMFormer REIN FADA
Builds on: REIN FADA DepthAnything V2 CLIP Mask2Former
Closest papers by meaning
- Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.90)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.90)
- Distilling Spectral Graph for Object-Context Aware Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)
- MaskViM: Domain Generalized Semantic Segmentation with State Space Models (AAAI 2025 · similarity 0.89)
- Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation (CVPR 2025 · similarity 0.89)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.