Research map / Vision-language and multimodal / CLIP and zero-shot
Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VPT-based parameter-efficient fine-tuning for deep metric learning, generating class-wise semantic proxies from learnable visual prompts integrated with a GRU, matching or beating full fine-tuning of ViT-S/16 with ~5% tunable parameters.
Problem
Adapting large pre-trained ViTs to deep metric learning (DML) retrieval tasks is costly with full fine-tuning and risks catastrophic forgetting; parameter-efficient methods for DML are under-explored, and randomly initialized proxies lack semantic information.
Approach
Learn class-conditioned visual prompts in a frozen ViT to produce semantic proxies, and accumulate them across batches with a ReLU-variant GRU (or EMA), then mix with ordinary learnable proxies as bias within a Proxy-Anchor loss framework.
Key results
- On CUB200 with ViT-S/16, VPTSP-G (B) reaches 86.6 R@1 and 52.7 MAP@R versus 85.5 R@1 and 51.1 MAP@R for full fine-tuning (PA), with 5.2% tunable parameters (1.56M of 30.2M).
- With ViT-B/16 at 512 dims, VPTSP-G reports 88.5 R@1 on CUB200 and 91.2 R@1 on CARS196, above the RS@K baseline's 89.5 on CARS196 in the table; on CUB200 RS@K is not reported.
- On SOP and InShop, the method is comparable to or below state-of-the-art full fine-tuning, with the authors acknowledging it does not outperform the latest SOTA on these datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic proxies to VPT (ablation) | VPT on ViT-S · CUB200 | R@1: 85.1 -> 86.6 | Helped |
| Use GRU rather than EMA to accumulate semantic proxies (ablation) | VPTSP · CUB200 | R@1: 85.4 -> 86.1 | Helped |
| Add BitFit to VPTSP-G (ablation) | VPTSP-G · CUB200 | R@1: 86.1 -> 86.6 | Helped |
| Add BitFit to VPTSP-G (ablation) | VPTSP-G · CARS196 | R@1: 87.1 -> 87.7 | Helped |
| Add BitFit to VPT (ablation) | VPT · CUB200 | R@1: 85.1 -> 85.7 | Helped |
| Add BitFit to Adapter (ablation) | Adapter · CUB200 | R@1: 83.1 -> 85.8 | Helped |
| Use class-based prompts to generate proxies instead of proxies from encoder samples (ablation) | VPTSP-G · CUB200 | MAP@R: 51.2 -> 52.7 | Helped |
| Generate proxies from the same trainable encoder prompts rather than the fixed encoder (ablation) | Proxy encoder · CUB200 | MAP@R: 50.5 -> 51.5 | Helped |
| Replace ReLU with Tanh in the GRU proxy accumulator (ablation) | GRU proxy accumulator · CUB200 | R@1: 86.1 -> 85.7 | Hurt |
| Use cross-attention instead of GRU to integrate semantic proxies (ablation) | Proxy integration · CUB200 | R@1: 86.1 -> 82.2 | Hurt |
| Combine VPT with Adapter (ablation) | VPT · DML benchmarks | performance | No clear effect |
| Increase the number of prompts beyond the dataset-dependent limit (ablation) | VPTSP · DML benchmarks | performance | Hurt |
Surprising findings
- Adding BitFit bias tuning to VPT and semantic proxies consistently improves results, and VPT alone with BitFit outperforms full fine-tuning on CUB200.
- Directly accumulating sample representations as semantic proxies without class-based prompts degrades later training due to semantic shift.
Limitations
- Does not outperform state-of-the-art full fine-tuning on SOP and InShop, attributed to the large number of classes making per-class proxy optimization hard.
- Memory for class-based prompts grows linearly with the number of classes.
- Proxy updates only occur for classes sampled in the mini-batch, requiring a bias term to compensate.
Experimental setup
Datasets: CUB-200-2011 CARS196 Stanford Online Products In-shop Clothes Retrieval iNaturalist ImageNet-21k
Benchmarks: CUB-200-2011 CARS196 Stanford Online Products In-shop Clothes Retrieval iNaturalist
Metrics: Recall@K MAP@R
Baselines: Full Fine-tuning (PA) Linear Probe Adapter BitFit VPT Hyp-ViT RS@K SAP ProxyNCA++ NIR
Builds on: Visual Prompt Tuning Proxy-Anchor loss BitFit gated recurrent unit
Closest papers by meaning
- DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers (CVPR 2025 · similarity 0.91)
- Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR 2024 · similarity 0.88)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.88)
- Towards Improved Proxy-Based Deep Metric Learning via Data-Augmented Domain Adaptation (AAAI 2024 · similarity 0.88)
- Potential Field Based Deep Metric Learning (CVPR 2025 · similarity 0.87)
- VIPAMIN: Visual Prompt Initialization via Embedding Selection and Subspace Expansion (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.