Research map / Vision-language and multimodal / CLIP and zero-shot
Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ECALP, a training-free graph label propagation method for zero-/few-shot CLIP adaptation that uses iterative propagation, dynamic graph expansion for streaming inference, and context-aware feature re-weighting, with fixed hyperparameters.
Problem
Zero- and few-shot adaptation of vision-language models often needs task-specific hyperparameter tuning and does not fully exploit the test-sample manifold; existing zero-shot label propagation (ZLaP) is computationally costly via closed-form inversion and uses cosine similarity that is biased by ir
Approach
Build a dynamic kNN graph over text prototypes, few-shot samples and streaming test samples, run a truncated iterative label propagation that resets prototype and labeled-node labels each step, and re-weight CLIP feature dimensions using variance statistics of text prompts and few-shot features when computing edge affinities.
Key results
- Zero-shot ViT-B/16 average over 11 fine-grained datasets: ECALP 70.54% vs. DMN 69.29% and ZLaP 67.72% (CLIP 63.87%).
- Out-of-distribution ImageNet-C average (ViT-B/16): ECALP 29.50% vs. TDA 28.34%, DMN 27.46%, CLIP 24.63%.
- Style-transfer average with ViT-B/16: ECALP 64.92% vs. TDA 63.89%, DMN 63.80%; ablation shows text re-weighting lifts ImageNet from 55.34% to 62.64% (RN50).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text-embedding edge reweighting to zero-shot label propagation (ablation) | ECALP-ZS with CLIP-RN50 · ImageNet | accuracy: 55.34% -> 62.64% | Helped |
| add text-embedding edge reweighting to zero-shot label propagation (ablation) | ECALP-ZS with CLIP-RN50 · DTD and UCF101 | accuracy: 49.76 -> 54.49; 61.17 -> 66.67 | Helped |
| add few-shot feature-distribution reweighting (ablation) | few-shot ECALP with CLIP-RN50 · ImageNet, DTD, and UCF101 | accuracy: 47.25 -> 63.47; 59.34 -> 65.13; 66.98 -> 74.18 | Helped |
| add text reweighting while retaining few-shot feature reweighting (ablation) | few-shot ECALP with CLIP-RN50 · ImageNet, DTD, and UCF101 | accuracy: 63.47 -> 63.20; 65.13 -> 65.37; 74.18 -> 74.44 | Mixed |
| add within-test-sample graph edges (ablation) | zero-shot ECALP with CLIP-RN50 · ImageNet, DTD, and UCF101 | accuracy: 62.12 -> 62.64; 52.42 -> 54.49; 63.97 -> 66.67 | Helped |
| add within-test-sample graph edges (ablation) | few-shot ECALP with CLIP-RN50 · ImageNet, DTD, and UCF101 | accuracy: 63.00 -> 63.20; 64.48 -> 65.37; 72.01 -> 74.44 | Helped |
| use label propagation instead of nearest-neighbor classification (ablation) | ECALP zero-shot with CLIP-RN50 · ImageNet | accuracy: 58.16% -> 55.34% | Hurt |
| use label propagation instead of nearest-neighbor classification (ablation) | ECALP zero-shot with CLIP-RN50 · DTD and UCF101 | accuracy | Helped |
| vary the test-to-prompt and test-to-test KNN counts; best reported setting is k_P = 3 and K_Du = 8 (ablation) | ECALP zero-shot with CLIP-RN50 · DTD | accuracy: 54.49% | Mixed |
| apply ECALP to the base vision-language model | BLIP, BLIP v2, and ALBEF · ImageNet | accuracy: 53.40% -> 58.16%; 41.22% -> 43.72%; 36.15% -> 40.31% | Helped |
Surprising findings
- Plain label propagation without re-weighting drops ImageNet accuracy below nearest-neighbor classification (58.16% CLIP-RN50 baseline vs. 55.34%).
- Combining ECALP with CoOp prompts underperforms standalone ECALP at 1-shot, attributed to overfitting of learned prompts in very low-shot regimes.
Limitations
- Few-shot gains are limited on datasets with many isolated sub-clusters per class such as OxfordPets and Food101 (acknowledged failure cases).
- Hyperparameters are fixed globally but some ablations (KNN connection counts, iteration T) show sensitivity to chosen values.
- ImageNet benefits less from few-shot samples due to high intra-class variation.
Experimental setup
Datasets: ImageNet Flowers102 DTD OxfordPets StanfordCars UCF101 Caltech101 Food101 SUN397 FGVCAircraft EuroSAT ImageNet-V2
Benchmarks: ImageNet Flowers102 DTD OxfordPets StanfordCars UCF101 Caltech101 Food101 SUN397 FGVCAircraft EuroSAT ImageNet-V2
Metrics: top-1 accuracy wall-clock testing time
Baselines: TPT DiffTPT CoOp CoCoOp TDA DMN ZLaP Tip-Adapter APE TIP-X CLIP-Adapter
Builds on: ZLaP Tip-Adapter DMN CLIP
Closest papers by meaning
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.89)
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.89)
- Label Propagation for Zero-shot Classification with Vision-Language Models (CVPR 2024 · similarity 0.89)
- p-Laplacian Adaptation for Generative Pre-trained Vision-Language Models (AAAI 2024 · similarity 0.89)
- Candidate Pseudolabel Learning: Enhancing Vision-Language Models by Prompt Tuning with Unlabeled Data (ICML 2024 · similarity 0.89)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.