Research map / Label-efficient and robust learning / Test-time adaptation
Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sparse Visual Domain Prompts (SVDP), which place sparse trainable pixel-wise prompts chosen by MC-dropout uncertainty and updated with uncertainty-dependent EMA, for test-time and continual test-time adaptation in semantic segmentation and depth estimation.
Problem
Visual domain prompts that overlay dense image-level patches occlude continuous spatial detail, harming dense prediction (semantic segmentation, depth estimation) under test-time and continual test-time adaptation.
Approach
Replace dense visual prompts with sparse pixel-wise prompt parameters (~0.1%) spread over the image, placed on high-uncertainty pixels (Domain Prompt Placement) and updated per sample with an EMA rate scaled by image-level uncertainty (Domain Prompt Updating), within a mean-teacher framework.
Key results
- On Cityscapes-to-ACDC TTA, SVDP reaches 60.1% mean mIoU versus 58.6% for CoTTA and 58.5% for VDP (source 56.7%).
- On Cityscapes-to-ACDC CTTA, SVDP reaches 61.1% mean mIoU, a +4.4 gain over the source model and about 2.5 points above CoTTA.
- On KITTI-to-DrivingStereo CTTA, SVDP achieves mean delta>1.25 of 0.741 versus 0.666 for VDP and 0.516 for CoTTA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SVDP instead of the source model without adaptation | Segformer-B5 · Cityscapes-to-ACDC TTA (four-domain mean) | Mean-mIoU: 56.7 -> 60.1±0.2 | Helped |
| Use SVDP instead of CoTTA | Segformer-B5 · Cityscapes-to-ACDC CTTA (three-round mean) | Mean mIoU: 58.6 -> 61.1±0.3 | Helped |
| Use SVDP instead of VDP | DPT · KITTI-to-Driving Stereo CTTA (four-domain mean) | Mean δ>1.25: 0.666 -> 0.741±0.03 | Helped |
| Add teacher-student structure (ablation) | KITTI-to-Driving Stereo CTTA ablation · KITTI-to-Driving Stereo CTTA | Abs Rel: 0.312 -> 0.249 | Helped |
| Add SVDP to teacher-student structure (ablation) | KITTI-to-Driving Stereo CTTA ablation · KITTI-to-Driving Stereo CTTA | Abs Rel: 0.249 -> 0.187 | Helped |
| Add DPP to teacher-student structure and SVDP (ablation) | KITTI-to-Driving Stereo CTTA ablation · KITTI-to-Driving Stereo CTTA | Abs Rel: 0.187 -> 0.169 | Helped |
| Add DPU to teacher-student structure and SVDP (ablation) | KITTI-to-Driving Stereo CTTA ablation · KITTI-to-Driving Stereo CTTA | Abs Rel: 0.187 -> 0.177 | Helped |
| Use the complete combination of TS, SVDP, DPP, and DPU (ablation) | KITTI-to-Driving Stereo CTTA ablation · KITTI-to-Driving Stereo CTTA | Abs Rel: 0.312 -> 0.162 | Helped |
| Increase SVDP pixel-parameter density, with density beyond 1e-3 (ablation) | SVDP depth estimation · KITTI-to-Driving Stereo CTTA | δ>1.25 | Hurt |
| Use per-sample uncertainty-based prompt EMA weights rather than a fixed best β (ablation) | SVDP depth estimation · KITTI-to-Driving Stereo CTTA | δ>1.25: 1.5% | Helped |
Surprising findings
- DePT's mean mIoU degrades over continual rounds (from 56.5 to 49.7 on the CTTA sequence), showing catastrophic forgetting of token-level prompts.
- Ablation shows that increasing prompt density beyond 1e-3 hurts depth performance because prompts occlude spatial detail.
Limitations
- Depth estimation TTA results are only in the appendix and the main CTTA evaluation is limited to two benchmarks.
- Sensitivity to prompt density requires tuning (best at 1e-3), and the EMA scaling factor theta is hand-set.
Experimental setup
Datasets: Cityscapes ACDC KITTI DrivingStereo
Benchmarks: Cityscapes-to-ACDC TTA Cityscapes-to-ACDC CTTA KITTI-to-DrivingStereo CTTA
Metrics: mIoU mAcc delta>1.25 AbsRel RMSE
Baselines: Source TENT CoTTA DePT VDP
Builds on: visual domain prompts (VDP) CoTTA Tent VPT DePT
Closest papers by meaning
- Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation (CVPR 2024 · similarity 0.90)
- Domain-Controlled Prompt Learning (AAAI 2024 · similarity 0.89)
- Enhancing Domain Adaptation through Prompt Gradient Alignment (NeurIPS 2024 · similarity 0.89)
- Preserving Clusters in Prompt Learning for Unsupervised Domain Adaptation (CVPR 2025 · similarity 0.88)
- Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach (CVPR 2024 · similarity 0.88)
- Prompt-Based Distribution Alignment for Unsupervised Domain Adaptation (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.