Research map / Vision-language and multimodal / CLIP and zero-shot
A Similarity Paradigm Through Textual Regularization Without Forgetting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPTR, a prompt-learning method for CLIP that uses optimal transport to regularize tuned text features toward multiple hand-crafted prompt features and a KL-based similarity paradigm between natural and adversarial alignment scores.
Problem
Prompt learning for vision-language models overfits downstream distributions, causing forgetting of hand-crafted prompt knowledge and poor generalization to unseen classes and domains.
Approach
Combine an optimal transport textual regularizer that aligns a single learnable prompt feature with multiple hand-crafted prompt features, and a KL-divergence similarity paradigm that matches natural and PGD-adversarial vision-language alignment scores, both built on hand-crafted CLIP prompts.
Key results
- Base-to-novel average HM of 80.61% over 11 datasets, above TCP (79.51%) and MaPLe (78.55%).
- Domain generalization average of 60.59% across ImageNet variants, versus 60.40% for PromptSRC re-implementation.
- Cross-dataset average of 66.95%, +1.8 over CLIP, versus +1.2 for TCP and MaPLe.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OT textual regularization to the baseline V-L prompting framework (ablation) | SPTR · Base-to-novel generalization, 11 datasets (average) | novel accuracy: 71.38 -> 74.90 | Helped |
| Add the similarity paradigm to SPTR with OT (ablation) | SPTR · Base-to-novel generalization, 11 datasets (average) | novel accuracy: 74.90 -> 76.76 | Helped |
| Add OT textual regularization and the similarity paradigm (ablation) | SPTR · Base-to-novel generalization, 11 datasets (average) | harmonic mean: 77.34 -> 80.61 | Helped |
| Replace OT with MSE regularization (ablation) | SPTR · Base-to-novel generalization, 11 datasets (average) | harmonic mean: 79.17 -> 80.61 | Hurt |
| Replace OT with L1 regularization (ablation) | SPTR · Base-to-novel generalization, 11 datasets (average) | harmonic mean: 80.09 -> 80.61 | Hurt |
| Replace OT with cosine regularization (ablation) | SPTR · Base-to-novel generalization, 11 datasets (average) | harmonic mean: 79.90 -> 80.61 | Hurt |
| Set embedding length to 4 rather than 1, 2, 6, or 8 (ablation) | SPTR · Caltech101 | harmonic mean: 95.13, 95.31, 95.87, 93.10, 95.33 | Mixed |
| Increase learning depth to 11 from 9 (ablation) | SPTR · Caltech101 | harmonic mean: 95.87 -> 95.33 | Hurt |
| Increase learning depth to 11 from 9 (ablation) | SPTR · Food101 | harmonic mean: 91.94 -> 91.10 | Hurt |
| Increase perturbation boundary from 1/255 to 4/255 or 8/255 (ablation) | SPTR · Caltech101 | harmonic mean: 95.87 -> 94.00; 95.87 -> 93.58 | Hurt |
| Increase perturbation boundary from 1/255 to 4/255 or 8/255 (ablation) | SPTR · Food101 | harmonic mean: 91.94 -> 90.88; 91.94 -> 89.45 | Hurt |
| Set textual regularization α to 0.3 instead of 0.1 or 0.5 (ablation) | SPTR · Caltech101 | harmonic mean: 93.00 -> 95.87; 95.18 -> 95.87 | Mixed |
Surprising findings
- The paper reports that the inference speed is relatively slow due to adversarial designs, a cost not offset by a large gain on all tasks.
Limitations
- Adversarial design makes inference relatively slow.
- Results on ViT architectures beyond ViT-B/16 are limited; authors plan further ViT validation.
- Excessive depth of learning degrades HM, indicating sensitivity to hyperparameters such as learning depth and epsilon.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT ImageNetV2
Metrics: accuracy harmonic mean (HM) base accuracy novel accuracy
Baselines: CLIP CoOp CoCoOp KgCoOp PLOT MaPLe PromptSRC TCP
Builds on: CoOp CoCoOp KgCoOp MaPLe PromptSRC PLOT
Closest papers by meaning
- PromptCoT: Align Prompt Distribution via Adapted Chain-of-Thought (CVPR 2024 · similarity 0.89)
- Compound Text-Guided Prompt Tuning via Image-Adaptive Cues (AAAI 2024 · similarity 0.89)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.89)
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.88)
- Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP (NeurIPS 2024 · similarity 0.88)
- Prompt Learning via Meta-Regularization (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.