Research map / Vision-language and multimodal / CLIP and zero-shot
Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies a "Low-Norm Effect" in CoOp soft prompts (shrinking certain prompt vectors can improve CLIP accuracy) and proposes Nemesis, a PUN/PAN norm-regularization loss that normalizes soft prompts for few-shot and generalization gains.
Problem
Soft-prompt tuning of vision-language models like CLIP has been studied extensively, but the effect of learned soft-prompt vector norms on VLM performance is largely unexplored.
Approach
Corruption experiments (REPLACE and RESCALE of individual prompt vectors) reveal that reducing norms at certain positions can improve accuracy (the Low-Norm Effect). Nemesis adds a norm regularizer: PUN applies it uniformly across positions, while PAN runs a pre-inference step each batch to select positions whose norm reduction improves predictions and normalizes only those.
Key results
- On ImageNet, CoOp+Nemesis (PUN) gains 2.06%, 3.84%, 2.6%, 1.16%, 0.38% over CoOp at 1, 2, 4, 8, 16 shots.
- In domain generalization (trained on ImageNet 16-shot), CoOp+Nemesis (PUN, omega=10) reaches 63.08 source and 56.78 on ImageNet-R vs CoOp's 62.70 and 54.60.
- Applying the PUN loss to PLOT improves average few-shot accuracy at every shot count (e.g., 83.75 to 83.96 at 16 shots for PUN, 84.15 for PAN).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Nemesis PUN/PAN normalization of soft prompts added to CoOp | CoOp · Few-shot recognition, 11 datasets | accuracy: large gains on ImageNet, OxfordPets, Food101, SUN397 | Helped |
| Nemesis PUN normalization (ω=10) added to CoOp | CoOp · Domain generalization from ImageNet | accuracy: +0.38 source, +0.46 V2, +1.70 Sketch, +0.96 A, +2.18 R (ω=10) | Helped |
| Larger normalization strength ω (ω=10 vs ω=1) in PUN (ablation) | CoOp+Nemesis PUN · Domain generalization from ImageNet | accuracy on target domains: Sketch 33.64→34.34, A 22.96→23.40, R 55.72→56.78 | Helped |
| Nemesis PAN normalization added to PLOT | PLOT · Few-shot recognition (Caltech101, DTD, Flowers102 average) | accuracy: 83.75 → 84.15 at 16 shots; improves at all shot settings on average | Helped |
| PAN position selection (vs. normalizing random positions without selection) (ablation) | CoOp+Nemesis PAN on FGVCAircraft · FGVCAircraft few-shot | accuracy: selection helps at N=16 16-shot (31.44 vs 31.12) and N=4 1-shot is lower (9.44 vs 10.04) | Mixed |
| Increasing number of corruption positions N in PAN (ablation) | CoOp+Nemesis PAN on FGVCAircraft · FGVCAircraft few-shot | accuracy: 1-shot 9.56 (CoOp) to 10.44 (N=16); 16-shot peaks at N=2 (32.46) and falls at N=16 (31.44) | Mixed |
| Mixing PUN and PAN losses (beta between 0 and 1) on ImageNet (ablation) | CoOp+Nemesis · ImageNet 1-shot and 16-shot | accuracy: best beta=0.3 at 1-shot (59.54); best beta=0.1 at 16-shot (74.39); PUN only (beta=1) 73.47 at 16-shot | Mixed |
| Applying PUN loss to visual prompt tuning and prefix-tuning (PEFT extension) | Other PEFT methods · Image classification (VPT) and paraphrase classification (prefix-tuning) | accuracy | Helped |
| Rescaling or replacing a soft prompt vector with lower norm at specific positions (Low-Norm Effect) (ablation) | CoOp corruption experiments · ImageNet 1-shot corruption | accuracy: Positions with rescale 0.5 at 1-shot: 14 of 16 positions improve | Helped |
| Increasing the norm of soft prompt vectors (rescale 2.0 or large variance replace) (ablation) | CoOp corruption experiments · 11 datasets corruption experiments | accuracy | Hurt |
| Nemesis gains reduced with fewer training shots (Low-Norm Effect frequency) | CoOp+Nemesis · Few-shot recognition | accuracy | No clear effect |
| Nemesis PAN with ω=20 added to CoCoOp | CoCoOp · Base-to-new generalization (new classes) | accuracy: +1.3% on new classes | Helped |
Surprising findings
- Reducing the norm of certain individual soft-prompt vectors occasionally improves CLIP accuracy, whereas increasing norms usually degrades it (Low-Norm Effect).
- Low-Norm Effect occurrence is negatively correlated with the number of shots, and it is concentrated in datasets such as ImageNet, OxfordPets and Food101.
Limitations
- Gains are inconsistent across datasets and shot settings; the Low-Norm Effect is not reliably manifest, and PAN is sometimes worse than PUN in generalization.
- PAN adds extra inference cost that grows with the number of corrupted positions N and the number of classes.
- Hyper-parameter omega must be tuned per dataset (e.g., 10 for ImageNet, 50 for OxfordPets and Food101), and the appropriate choice is left open.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNetV2
Metrics: top-1 accuracy harmonic mean
Baselines: CoOp CoCoOp PLOT
Builds on: CoOp CoCoOp PLOT CLIP
Closest papers by meaning
- Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.90)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.90)
- ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models (CVPR 2024 · similarity 0.89)
- PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models (CVPR 2025 · similarity 0.89)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.89)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.