Research map / Learning theory and algorithms / Kernels and regression theory
Understanding prompt engineering may not require rethinking generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that discrete prompt engineering for CLIP admits tight PAC-Bayes generalization bounds when the prior is a language model (LLaMA-7B) over prompt tokens, with the bound often within a few points of test error on ImageNet and CIFAR.
Problem
Prompted vision-language models (zero-shot classifiers built from manually or greedily engineered prompts) show little overfitting even when prompts are tuned on training data, and existing generalization bounds for deep learning are loose or vacuous, so this behavior lacks theoretical explanation.
Approach
Treat discrete text prompts as a small hypothesis class, use a pretrained language model as the PAC-Bayes prior over tokens, and use a point-mass posterior so the KL divergence reduces to the negative log-likelihood of the prompt under the LM, yielding a deterministic bound. Also proposes greedy token search with LM-pruned candidates and an SRM variant penalizing this KL term.
Key results
- The PAC-Bayes (prompt) bound on ImageNet with handcrafted L-14 prompts is 0.319 versus 0.256 test error, far tighter than prior bounds of 0.930 and 0.409 for data-dependent approaches.
- On CIFAR-10 the greedy L-14 prompt has 0.028 test error with a PAC-Bayes bound of 0.063, and on CIFAR-100 0.266 bound versus 0.180 test error.
- Greedy prompts fit random labels poorly: training and test accuracy both drop monotonically as label noise increases on CIFAR-10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PAC-Bayes bounds with an LLM prior instead of uniform convergence bounds | Greedy prompts · CIFAR-10, CIFAR-100, fMoW, OfficeHome | generalization bound: CIFAR-10: 0.154 -> 0.086; CIFAR-100: 0.537 -> 0.317; fMoW: 0.807 -> 0.667; OfficeHome: 0.635 -> 0.281 | Helped |
| Use CLIP ViT-L/14 instead of CLIP ViT-B/16 | Greedy · CIFAR-10 | test error: 0.060 -> 0.028 | Helped |
| Use CLIP ViT-L/14 instead of CLIP ViT-B/16 | Greedy · CIFAR-100 | test error: 0.255 -> 0.180 | Helped |
| Use CLIP ViT-L/14 instead of CLIP ViT-B/16 | Greedy · fMoW | test error: 0.621 -> 0.547 | Helped |
| Use CLIP ViT-L/14 instead of CLIP ViT-B/16 | Greedy · OfficeHome | test error: 0.150 -> 0.115 | Helped |
| Replace Greedy prompts with handcrafted prompts | CLIP ViT-L/14 · CIFAR-10 | test error: 0.028 -> 0.040 | Hurt |
| Replace Greedy prompts with handcrafted prompts | CLIP ViT-L/14 · CIFAR-100 | test error: 0.180 -> 0.221 | Hurt |
| Replace Greedy prompts with handcrafted prompts | CLIP ViT-L/14 · fMoW | test error: 0.547 -> 0.402 | Mixed |
| Replace Greedy prompts with handcrafted prompts | CLIP ViT-L/14 · OfficeHome | test error: 0.115 -> 0.928 | Hurt |
| Use a linear probe instead of discrete prompt search | CLIP features with 20 samples per class · CIFAR-10 | test error: 0.138 -> 0.038 | Helped |
| Use a linear probe instead of discrete prompt search | CLIP features with 20 samples per class · CIFAR-100 | test error: 0.367 -> 0.198 | Helped |
| Penalize search with the PAC-Bayes complexity term (SRM) (ablation) | Greedy prompts · CIFAR-10 | generalization bound | Helped |
Surprising findings
- Greedy prompt search does not fit randomly labeled data, in contrast to conventional deep networks that can memorize random labels.
- A linear probe on CLIP features achieves slightly better accuracy with 20 samples per class but yields a vacuous bound (2.591 PAC-Bayes-20).
Limitations
- The bounds assume the image encoder (CLIP) was not trained on the prompt-engineering training data, which cannot be verified for CLIP's undisclosed training data.
- The bounds do not explain the generalization of the underlying pretrained model itself, only of the prompt selection step.
- With only 20 samples per class, PAC-Bayes bounds are loose, though non-vacuous.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet fMoW OfficeHome
Benchmarks: CIFAR-10 CIFAR-100 ImageNet fMoW OfficeHome
Metrics: test error PAC-Bayes bound uniform convergence bound
Baselines: Uniform convergence bound Linear probe Zhou et al. PAC-Bayes bound (data-independent) Data-dependent PAC-Bayes bounds Handcrafted CLIP prompts
Builds on: PAC-Bayes bounds CLIP Wise-FT McAllester PAC-Bayes bound
Closest papers by meaning
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.90)
- Any-Shift Prompting for Generalization over Distributions (CVPR 2024 · similarity 0.89)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.88)
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.88)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.88)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.