Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
OTTER: Effortless Label Distribution Adaptation of Zero-shot Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OTTER, a tuning-free inference-time method that adjusts zero-shot classifier predictions via optimal transport to match an estimated downstream label distribution, with theory on Bayes-optimality and perturbation bounds.
Problem
Zero-shot models are biased by label distribution of their web-scale pretraining data, hurting accuracy when downstream label distribution differs; existing label-shift fixes need labeled downstream data or the true pretraining label distribution.
Approach
Treat zero-shot prediction rebalancing as an optimal transport problem: transport the batch of inputs onto the estimated target label marginal using cost -log of the model's scores, so predictions match the label specification without source-data access, training, or hyperparameter tuning. An online variant (R-OTTER) learns reweighting factors from OTTER pseudo-labels.
Key results
- OTTER with true label distribution improves zero-shot CLIP ViT-B/16 image classification by 4.8% on average (17 image datasets) and BERT text classification by about 15.5% on average, beating Prior Matching in 17 of 21 datasets.
- With BBSE-estimated distributions in 10-shot setting, OTTER gives mild gains over linear probing on images and large gains on small-K text tasks.
- On LLM multiple-choice selection bias, OTTER raises accuracy by up to 10.6% and lowers recall standard deviation by up to 25.7% across Llama-2 and Vicuna models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply OTTER using the true downstream label distribution | CLIP ViT-B/16 image classification · Image classification datasets | accuracy (%): 4.8% | Helped |
| Apply OTTER using the true downstream label distribution | BERT text classification · Text classification datasets | accuracy (%): 15.5% | Helped |
| Replace batched OTTER with learned reweighting (ablation) | OTTER · CIFAR10 | accuracy (%): 91.7 -> 88.4 | Hurt |
| Add label hierarchy information (ablation) | OTTER with CLIP RN50 · CIFAR100 fish/tree subset | accuracy (%): 38.5 ($\pm 4.9$) -> 43.6 ($\pm 3.1$) | Helped |
| Use uniform-balance OTTER with shuffled answer options | Llama-2-7B · ARC-Challenge | accuracy: 36.0 -> 45.5 | Helped |
| Use uniform-balance OTTER with shuffled answer options | Llama-2-7B · CommonsenseQA | accuracy: 31.9 -> 42.7 | Helped |
| Use uniform-balance OTTER with shuffled answer options | Llama-2-7B · MMLU | accuracy: 36.1 -> 40.0 | Helped |
| Use uniform-balance OTTER with shuffled answer options | Llama-2-7B · ARC-Challenge | RStd: 27.4 -> 1.7 | Helped |
Surprising findings
- Prior Matching's performance is highly sensitive to temperature and learning rate, and its hyperparameter tuning fails on class-imbalanced datasets such as Caltech256, SUN397, and ImageNet-r.
- Combining OTTER with BBSE estimates from zero-shot scores often hurts image accuracy (e.g., CIFAR100 falls to 59.1 from 63.8 zero-shot), so accuracy depends heavily on estimation quality.
Limitations
- Requires a batch of inference points and an estimate of the downstream label distribution; estimation error degrades accuracy.
- Theoretical guarantees assume calibrated scores and no ties; the Hoffman constant is not computed or bounded.
- Label-distribution estimation via BBSE relies on label shift assumptions that may be violated in real data.
Experimental setup
Datasets: CIFAR10 CIFAR100 Caltech101 Caltech256 Food101 STL10 SUN397 Flowers102 EuroSAT Oxford-IIIT Pet Stanford Cars DTD
Benchmarks: CIFAR10 CIFAR100 Caltech101 Caltech256 Food101 STL10 SUN397 Flowers102 EuroSAT Oxford-IIIT Pet Stanford Cars DTD
Metrics: accuracy standard deviation of recalls
Baselines: Zero-shot Prior Matching Linear probing BBSE
Builds on: CLIP Prior Matching Black Box Shift Estimation
Closest papers by meaning
- Zero-Shot Robustification of Zero-Shot Models (ICLR 2024 · similarity 0.89)
- Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting (NeurIPS 2024 · similarity 0.88)
- No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance (NeurIPS 2024 · similarity 0.87)
- Let Go of Your Labels with Unsupervised Transfer (ICML 2024 · similarity 0.87)
- Class Distribution Shifts in Zero-Shot Learning: Learning Robust Representations (NeurIPS 2024 · similarity 0.87)
- Data Distribution Distilled Generative Model for Generalized Zero-Shot Recognition (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.