Research map / Vision-language and multimodal / CLIP and zero-shot
Label Propagation for Zero-shot Classification with Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZLaP, a training-free label propagation method over bimodal image-text graphs that improves zero-shot CLIP classification in transductive and inductive setups using unlabeled images.
Problem
Zero-shot classification with vision-language models can be improved when unlabeled target-domain images are available, but naive graph-based label propagation fails due to the modality gap between image and text embeddings, and inductive inference with LP is expensive.
Approach
Adapts label propagation to bimodal graphs by performing kNN search separately within image nodes and from images to text class nodes, balances cross-modal edges with a power function, and enables fast inductive inference via a dual solution and sparsification of the inverse Laplacian's class columns.
Key results
- ZLaP with InMaP proxies on CLIP ViT-B/16 reaches about +5% average accuracy over CLIP in transductive setup (74.8 vs 71.9 with CuPL prompts; ~70.7 vs 67.3 with 7 templates).
- On MS-COCO multi-label classification, inductive ZLaP improves CLIP mAP from 50.8% to 56.8%.
- Separate kNN search across modalities is crucial: without it, ImageNet transductive accuracy drops to 0.1% versus 61.8% with adjustments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ZLaP to CLIP | CLIP ResNet50 · 14 datasets (average), transductive | accuracy: 57.3 -> 60.0 | Helped |
| Apply ZLaP to InMaP class representations | CLIP ResNet50 · 14 datasets (average), transductive | accuracy: 57.3 -> 61.3 | Helped |
| Apply ZLaP to CLIP | CLIP ViT-B/16 · 14 datasets (average), transductive | accuracy: 66.8 -> 68.9 | Helped |
| Apply ZLaP with a sparsified Y matrix | CLIP ResNet50 · 14 datasets (average), inductive | accuracy: 58.7 -> 59.5 | Helped |
| Apply ZLaP using CuPL prompts | CLIP · 12 datasets, transductive | accuracy: 63.0 -> 64.6 | Helped |
| Apply ZLaP | BLIP · ImageNet, transductive | accuracy: 54.6 -> 59.6 | Helped |
| Apply ZLaP | ALBEF · ImageNet, inductive | accuracy: 36.0 -> 46.8 | Helped |
| Use separate modality-specific kNN search instead of joint kNN search (ablation) | ZLaP · ImageNet, transductive | accuracy: 0.1 -> 50.2 | Helped |
| Add the power function for cross-modal similarities (ablation) | ZLaP with separate modality-specific search · ImageNet, transductive | accuracy: 50.2 -> 61.8 | Helped |
| Use separate modality-specific kNN search instead of joint kNN search (ablation) | ZLaP · DTD, transductive | accuracy: 2.1 -> 41.9 | Helped |
| Use separate modality-specific kNN search instead of joint kNN search (ablation) | ZLaP · ImageNet, inductive | accuracy: 0.1 -> 60.8 | Helped |
| Add the power function for cross-modal similarities (ablation) | ZLaP with separate modality-specific search · ImageNet, inductive | accuracy: 60.8 -> 62.2 | Helped |
Surprising findings
- Naive label propagation on bimodal CLIP graphs fails almost completely (near-zero accuracy) due to the modality gap, so the bimodal adjustments are essential.
- Random selection of web-crawled LAION-400M images fails to improve over CLIP, while proximity-based selection helps.
Limitations
- Gains over InMaP alone are smaller on average; ZLaP on CLIP underperforms InMaP in the inductive setting.
- Requires a set of unlabeled images from the target distribution; random web-crawled unlabeled data does not help.
- Hyperparameters (k, gamma, alpha) fixed across datasets but chosen per backbone/proxy setting.
Experimental setup
Datasets: ImageNet DTD EuroSAT FGVC-Aircraft Oxford Flowers 102 Food-101 Oxford-IIIT Pet SUN397 Stanford Cars Caltech101 UCF101 CIFAR10
Benchmarks: ImageNet DTD EuroSAT FGVC-Aircraft Oxford Flowers 102 Food-101 Oxford-IIIT Pet SUN397 Stanford Cars Caltech101 UCF101 CIFAR10
Metrics: accuracy mAP
Baselines: CLIP TPT CLIP-DN InMaP BLIP ALBEF EVA-CLIP
Builds on: label propagation CLIP InMaP CLIP-DN SuS-X
Closest papers by meaning
- Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model (ICLR 2025 · similarity 0.89)
- Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions (CVPR 2024 · similarity 0.89)
- Super-Class Guided Transformer for Zero-Shot Attribute Classification (AAAI 2025 · similarity 0.88)
- ZeroDiff: Solidified Visual-semantic Correlation in Zero-Shot Learning (ICLR 2025 · similarity 0.88)
- S3A: Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment (AAAI 2024 · similarity 0.87)
- Epsilon: Exploring Comprehensive Visual-Semantic Projection for Multi-Label Zero-Shot Learning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.