Research map / Label-efficient and robust learning / Domain adaptation and generalization
ProS: Prompting-to-simulate Generalized knowledge for Universal Cross-Domain Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProS, a prompt-tuning method for Universal Cross-Domain Retrieval that learns domain and semantic prompt units and then trains a context-aware simulator to generate content-aware dynamic prompts for frozen CLIP.
Problem
Universal Cross-Domain Retrieval (UCDR) requires generalizing to unseen domains and unseen categories at test time; existing prompt tuning methods applied to CLIP do not handle both domain shift and semantic shift well.
Approach
Two-stage prompt learning: first learn separate domain and semantic prompt units via a mask-and-align objective, then train a Content-aware Prompt Simulator (a 2-layer ViT) under simulated unseen-domain/category masking to dynamically generate content-aware dynamic prompts for the frozen CLIP image encoder.
Key results
- On DomainNet UCDR, ProS reaches 0.6052 average mAP@200 versus 0.5720 for VPT and 0.5229 for CLIP-Full.
- On TU-Berlin UcCDR, ProS achieves 0.6675 mAP@all, versus 0.5574 for VPT and 0.4715 for SASA.
- Ablation on Infograph (mixed gallery) shows removing the content-aware simulator drops mAP@200 from 0.5219 to 0.5053, and removing the CLS token drops it to 0.4601.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ProS instead of VPT | CLIP ViT-B/32 · DomainNet UCDR, Unseen Gallery average | mAP@200: 0.5720 -> 0.6052 | Helped |
| Use ProS instead of VPT | CLIP ViT-B/32 · DomainNet UCDR, Mixed Gallery average | mAP@200: 0.5796 -> 0.5546 | Hurt |
| Use ProS instead of VPT | CLIP ViT-B/32 · DomainNet UdCDR average | mAP@200: 0.5796 -> 0.6332 | Helped |
| Use ProS instead of SASA | Sketchy UcCDR · Sketchy | mAP@200: 0.6910 -> 0.6991 | Helped |
| Use ProS instead of CLIP-Full | TU-Berlin UcCDR · TU-Berlin | mAP@all: 0.6076 -> 0.6675 | Helped |
| Add Semantic Prompt Units (ablation) | ProS · DomainNet UCDR, Infograph query, Unseen Gallery | mAP@200: 0.5529 -> 0.5798 | Helped |
| Add Domain Prompt Units (ablation) | ProS · DomainNet UCDR, Infograph query, Mixed Gallery | mAP@200: 0.4963 -> 0.5219 | Helped |
| Add mask operation (ablation) | ProS · DomainNet UCDR, Infograph query, Mixed Gallery | mAP@200: 0.5092 -> 0.5219 | Helped |
| Add Content-aware Prompt Simulator (ablation) | ProS · DomainNet UCDR, Infograph query, Mixed Gallery | mAP@200: 0.5053 -> 0.5219 | Helped |
| Add CLS token (ablation) | ProS · DomainNet UCDR, Infograph query, Mixed Gallery | mAP@200: 0.4601 -> 0.5219 | Helped |
| Use 1 CaPS Transformer layer instead of 2 (ablation) | ProS · DomainNet UCDR, Infograph query, Unseen Gallery | mAP@200: 0.5798 -> 0.5752 | Hurt |
| Use 3 CaPS Transformer layers instead of 2 (ablation) | ProS · DomainNet UCDR, Infograph query, Unseen Gallery | mAP@200: 0.5798 -> 0.5790 | Hurt |
Surprising findings
- Full fine-tuning of CLIP is not uniformly better than prompt tuning baselines; CoOp scores below CLIP-Full on some UcCDR settings (e.g., Sketchy mAP@200 0.5074 vs 0.6553).
- Prompt tuning with ViT-based SASA on Sketchy outperforms all CLIP-based methods, with the authors attributing CLIP's low Sketchy mAP to visual backbone limitations.
Limitations
- Performance on Sketch and Quickdraw query domains remains relatively low due to dataset quality issues such as missing color and texture information.
- Performance drops when the gallery mixes seen and unseen classes due to interference from seen classes.
Experimental setup
Datasets: DomainNet Sketchy TU-Berlin
Benchmarks: DomainNet Sketchy TU-Berlin
Metrics: mAP@200 Prec@200 mAP@all Prec@100
Baselines: SnMpNet SASA CLIP-Full CoOp VPT
Builds on: CLIP VPT CoOp CoCoOp
Closest papers by meaning
- Domain-Controlled Prompt Learning (AAAI 2024 · similarity 0.86)
- Retrieval Across Any Domains via Large-scale Pre-trained Model (ICML 2024 · similarity 0.86)
- Semantic Feature Learning for Universal Unsupervised Cross-Domain Retrieval (NeurIPS 2024 · similarity 0.85)
- Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models (ICLR 2025 · similarity 0.85)
- Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation (CVPR 2024 · similarity 0.85)
- Prompt-Based Distribution Alignment for Unsupervised Domain Adaptation (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.