Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Conformal Prediction for Zero-Shot Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies split conformal prediction to CLIP zero-shot classifiers and proposes Conf-OT, a training-free transductive optimal-transport calibration of logits that shrinks prediction sets while keeping coverage.
Problem
Split conformal prediction on zero-shot vision-language models yields inefficient (large) prediction sets because of the domain gap between inaccessible pre-training data and the target task, and adapting with the calibration set breaks exchangeability.
Approach
Conf-OT solves a Sinkhorn optimal transport problem jointly over calibration and query similarity logits, with the label marginal constrained to the calibration label distribution, producing soft codes on which standard non-conformity scores are computed; this is unsupervised and transductive so exchangeability is preserved.
Key results
- Conf-OT reduces average set size by about 20% relative to base LAC, APS and RAPS on CLIP ViT-B/16 at alpha=0.10 (e.g., LAC 5.52 to 4.40) while keeping coverage near 0.90.
- Conf-OT improves Top-1 accuracy by 2.9 points for CLIP ViT-B/16 (63.8 to 66.7) averaged over 15 datasets.
- Conf-OT runs about 15x faster than transductive baselines such as TIM and TransCLIP (0.61-0.74 s vs 11-12 s on the benchmark).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Conf-OT to zero-shot predictions | CLIP ViT-B/16 with LAC · 15 datasets (average) | average set size: 5.52 -> 4.40 | Helped |
| Apply Conf-OT to zero-shot predictions | CLIP ViT-B/16 with LAC · 15 datasets (average) | Top-1 accuracy: 63.8 -> 66.7 | Helped |
| Apply Conf-OT to zero-shot predictions | CLIP ViT-B/16 with APS · 15 datasets (average), α = 0.10 | average set size: 9.87 -> 7.64 | Helped |
| Apply Conf-OT to zero-shot predictions | CLIP ViT-B/16 with RAPS · 15 datasets (average), α = 0.10 | average set size: 8.12 -> 6.68 | Helped |
| Use TIMKL(ĉm||uK) adaptation | CLIP ViT-B/16 with APS · 15 datasets (average), α = 0.10 | average set size: 9.87 -> 7.24 | Helped |
| Use TransCLIP adaptation | CLIP ViT-B/16 with LAC · 15 datasets (average), α = 0.10 | average set size: 5.52 -> 5.76 | Hurt |
| Use TransCLIP adaptation | CLIP ViT-B/16 with APS · 15 datasets (average), α = 0.10 | coverage: 0.900 -> 0.892 | Hurt |
| Use the calibration-set label-marginal prior instead of a uniform prior (ablation) | Conf-OT with LAC · 15 datasets (average), α = 0.10 | average set size: 5.32 -> 4.40 | Helped |
| Use the calibration-set label-marginal prior instead of a uniform prior (ablation) | Conf-OT with LAC · 15 datasets (average), α = 0.10 | Top-1 accuracy: 65.5 -> 66.7 | Helped |
| Use the calibration-set label-marginal prior instead of a uniform prior (ablation) | Conf-OT with APS · 15 datasets (average), α = 0.10 | average set size: 8.72 -> 7.64 | Helped |
| Use the calibration-set label-marginal prior instead of a uniform prior (ablation) | Conf-OT with RAPS · 15 datasets (average), α = 0.10 | average set size: 7.57 -> 6.68 | Helped |
| Set the Conf-OT entropic temperature below 1 (ablation) | Conf-OT with APS or RAPS · 15 datasets, α = 0.10 | average set size | Helped |
Surprising findings
- TIM and TransCLIP deteriorate set sizes when used with the LAC score despite improving accuracy.
- Improvements in set size are not correlated with accuracy gains across datasets, indicating discriminative and conformal efficiency behave differently.
Limitations
- Coverage guarantees hold only under data exchangeability between calibration and test sets.
- Adds some inference-time overhead compared with the inductive base setting.
- Hyper-parameters (entropic weight, Sinkhorn iterations) fixed across tasks without tuning.
Experimental setup
Datasets: ImageNet ImageNet-A ImageNetV2 ImageNet-R ImageNet-Sketch SUN397 FGVCAircraft EuroSAT StanfordCars Food101 OxfordPets Flowers102
Benchmarks: ImageNet ImageNet-A ImageNetV2 ImageNet-R ImageNet-Sketch SUN397 FGVCAircraft EuroSAT StanfordCars Food101 OxfordPets Flowers102
Metrics: Top-1 accuracy marginal coverage average set size class-conditional coverage gap
Baselines: LAC APS RAPS TIM (KL-regularized) TransCLIP Base SCP
Builds on: split conformal prediction LAC APS RAPS TIM TransCLIP
Closest papers by meaning
- Scaling Full Conformal Image Classifiers (NeurIPS 2026 · similarity 0.90)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.88)
- Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models (ICLR 2024 · similarity 0.88)
- Transductive Zero-Shot and Few-Shot CLIP (CVPR 2024 · similarity 0.88)
- Boosting Vision-Language Models with Transduction (NeurIPS 2024 · similarity 0.87)
- Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correcting (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.