Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
A Generalization Theory for Zero-Shot Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a learning-theoretic framework for zero-shot prediction (ZSP) that decomposes downstream error into prompt bias, residual dependence, and estimation error, with RKHS finite-sample bounds and simulated/image experiments on CLIP-style models.
Problem
Theoretical understanding of zero-shot prediction from self-supervised/multimodal contrastive pre-training plus prompting, comparing it to direct supervised prediction with dependence on pre-training, evaluation, and prompt distributions.
Approach
Models ZSP as an indirect two-stage predictor through a caption variable using a latent caption model; shows the gap to the direct predictor is bounded by residual dependence (conditional mean square contingency of X and Y given Z) plus prompt bias, and gives two estimator classes (conditional mean and information density) with finite-sample bounds.
Key results
- In a simulated Gaussian binary task, as residual dependence goes to zero the indirect predictor, CLIP and VICReg approach the direct predictor's accuracy.
- Unbiased prompting with ImageNet-Captions outperforms 18 default templates at about M=10 prompts per class, with almost 15% absolute top-1 gain for ResNet50 on Sub-Task 1.
- LLM-generated class-conditional prompts (CuPL with LLaMA 3) give nearly 20% top-5 gain over DTD templates for ResNet50 and ViT-B/32.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| sample prompts from P_{Y,Z} rather than use the 18 default templates | ResNet50 zero-shot classification · ImageNet-Captions Sub-Task 1 | top-1 accuracy: almost 15% absolute increase | Helped |
| use LLM-generated class-conditional captions instead of templates | CuPL · DTD | top-5 accuracy: nearly 20% increase | Helped |
| increase the number of unbiased prompts per class | zero-shot classification · ImageNet-Captions tasks | top-1 accuracy | Helped |
| use the single default template instead of class-conditional LLM prompts | zero-shot classification · Flowers 102 | accuracy | Helped |
| increase the number of LLM-generated prompts per class | class-conditional prompting · Flowers 102 and FGVC Aircraft | top-k accuracy | Helped |
| increase the number of LLM-generated prompts per class | class-conditional prompting · DTD | top-k accuracy | Helped |
| reduce conditional dependence between X and Y given Z | indirect predictor, CLIP, and VICReg · simulated binary classification | classification accuracy | Helped |
Surprising findings
- Default template sets can be outperformed by a single well-chosen template, e.g. Flowers 102's one default prompt often beats the LLM class-conditional approach on average.
Limitations
- Theoretical guarantees rely on RKHS function classes and assumptions (e.g. well-specification, bounded r) that may not match deep encoders.
- Experiments use lightweight CLIP encoders on a limited set of datasets and simulated data.
- Kernel analysis is used as insight, not an interpretation of foundation models as kernel methods.
Experimental setup
Datasets: YFCC15M LAION COCO subset DataComp medium ImageNet-Captions
Benchmarks: DTD Flowers 102 FGVC Aircraft SUN397 ImageNet-1k
Metrics: top-k zero-shot accuracy
Baselines: direct predictor CLIP VICReg template-based prompting
Builds on: CLIP CuPL multi-view redundancy theory spectral regularization kernel learning
Closest papers by meaning
- Zero-Shot Robustification of Zero-Shot Models (ICLR 2024 · similarity 0.87)
- Understanding prompt engineering may not require rethinking generalization (ICLR 2024 · similarity 0.87)
- Zero-Shot Task Adaptation with Relevant Feature Information (AAAI 2024 · similarity 0.87)
- Sharpness-Aware Data Generation for Zero-shot Quantization (ICML 2024 · similarity 0.86)
- Class Distribution Shifts in Zero-Shot Learning: Learning Robust Representations (NeurIPS 2024 · similarity 0.86)
- Learning to Route Among Specialized Experts for Zero-Shot Generalization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.