Research map / Detection and segmentation / Open-set and few-shot detection
Better to Teach than to Give: Domain Generalized Semantic Segmentation via Agent Queries with Diffusion Model Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QueryDiff, which uses learnable agent queries aggregated from segmentation features and guided by frozen Stable Diffusion features (via a diffusion consistency loss) to improve domain generalized semantic segmentation.
Problem
Domain generalized semantic segmentation must generalize from a labeled synthetic or real source domain to unseen target domains, but existing domain randomization and diffusion-based data generation fail to capture the underlying scene distribution and are costly.
Approach
Use learnable agent queries as an interface between the segmentation backbone and frozen diffusion features (no diffusion at inference); a diffusion consistency loss strips low-level appearance detail from diffusion features so the queries learn scene-level semantic distribution, and the queries then refine decoder features before mask prediction.
Key results
- Under G→C,B,M with DINOv2-L backbone, QueryDiff reaches 66.7 average mIoU vs 64.3 for Rein (+2.4).
- With MiT-B5 under G→C,B,M, QueryDiff averages 60.2 vs 55.9 for HRDA and 57.8 for DGInStyle.
- Under C→ACDC normal-to-adverse with DINOv2-L, QueryDiff averages 79.9 mIoU vs 77.6 for Rein.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add agent queries and Lsup to Rein (ablation) | Rein with DINOv2-L · GTA5→Cityscapes, BDD100K, Mapillary (Avg.) | mIoU: 64.3 -> 65.4 | Helped |
| Add Ldist to agent queries + Lsup (ablation) | Rein with DINOv2-L · GTA5→Cityscapes, BDD100K, Mapillary (Avg.) | mIoU: 65.4 -> 66.1 | Helped |
| Add agent queries, Lsup, and Ldist to Rein (ablation) | Rein with DINOv2-L · GTA5→Cityscapes, BDD100K, Mapillary (Avg.) | mIoU: 64.3 -> 66.7 | Helped |
| Use timesteps tw=0 and ts=50 (ablation) | QueryDiff · GTA5→Cityscapes, BDD100K, Mapillary (Avg.) | mIoU: 66.7 -> 66.2 | Hurt |
| Use timesteps tw=50 and ts=100 (ablation) | QueryDiff · GTA5→Cityscapes, BDD100K, Mapillary (Avg.) | mIoU: 66.7 -> 66.5 | Hurt |
| Use timesteps tw=75 and ts=100 (ablation) | QueryDiff · GTA5→Cityscapes, BDD100K, Mapillary (Avg.) | mIoU: 66.7 -> 66.1 | Hurt |
| Use timesteps tw=100 and ts=200 (ablation) | QueryDiff · GTA5→Cityscapes, BDD100K, Mapillary (Avg.) | mIoU: 66.7 -> 65.9 | Hurt |
| Use timesteps tw=0 and ts=200 (ablation) | QueryDiff · GTA5→Cityscapes, BDD100K, Mapillary (Avg.) | mIoU: 66.7 -> 65.6 | Hurt |
| Replace Stable Diffusion 2.1 with Stable Diffusion 1.4 (ablation) | QueryDiff · GTA5→Cityscapes, BDD100K, Mapillary | mIoU: 66.7 -> 66.4 | Hurt |
| Replace Stable Diffusion 2.1 with Stable Diffusion 1.5 (ablation) | QueryDiff · GTA5→Cityscapes, BDD100K, Mapillary | mIoU: 66.7 -> 66.6 | Hurt |
| Use QueryDiff instead of Rein | DINOv2-L · GTA5→Cityscapes, BDD100K, Mapillary (Avg.) | mIoU: 64.3 -> 66.7 | Helped |
| Use QueryDiff instead of Rein | DINOv2-L · Cityscapes→BDD100K, Mapillary (Avg.) | mIoU: 68.8 -> 71.0 | Helped |
Limitations
- Relies on a frozen Stable Diffusion model and its feature-extraction timesteps; hyperparameters (tw, ts, r) were tuned empirically.
- Gains over Rein are modest on some settings (e.g., +1.1 mIoU from agent queries alone in ablation).
Experimental setup
Datasets: GTA5 Cityscapes BDD100K Mapillary ACDC
Benchmarks: Cityscapes BDD100K Mapillary ACDC
Metrics: mIoU
Baselines: Rein HGFormer DGInStyle CLOUDS HRDA FAMix BlindNet DIDEX WildNet SHADE SAW TLDR
Builds on: Mask2Former Rein Stable Diffusion DGInStyle
Closest papers by meaning
- Collaborating Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.89)
- Stronger Fewer & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation (CVPR 2024 · similarity 0.89)
- Exploring Semantic Consistency and Style Diversity for Domain Generalized Semantic Segmentation (AAAI 2025 · similarity 0.88)
- A Simple Recipe for Language-guided Domain Generalized Segmentation (CVPR 2024 · similarity 0.88)
- MaskViM: Domain Generalized Semantic Segmentation with State Space Models (AAAI 2025 · similarity 0.88)
- Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.