Research map / Vision-language and multimodal / CLIP and zero-shot
Boosting Vision-Language Models with Transduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TransCLIP, a plug-and-play transductive objective for vision-language models that combines GMM clustering, a Laplacian regularizer, and a text-guided KL penalty, optimized with block majorize-minimize for black-box zero- and few-shot inference.
Problem
Inductive zero- and few-shot adaptation of vision-language models predicts each test sample independently; existing transductive few-shot methods built for vision-only models perform poorly on VLMs because they ignore textual class knowledge.
Approach
Joint transductive inference over unlabeled test samples via a GMM clustering objective with a Laplacian regularizer, plus a KL divergence penalty tying assignments to CLIP's zero-shot text-based predictions; optimized with a Block Majorize-Minimize procedure with decoupled, parallel per-sample z-updates.
Key results
- TransCLIP-ZS improves CLIP-ViT-B/16 zero-shot average over 11 datasets from 65.3% to 70.3% (ImageNet 66.6 to 70.3).
- TransCLIP-FS at 4-shot reaches 75.9% average vs. 67.0-71.3% for prior transductive few-shot methods (LaplacianShot, TIM, PT-MAP), and removing the text KL term drops 1-shot average from 73.2 to 50.2.
- On EVA-CLIP-8B, TransCLIP-ZS raises the 11-dataset average from 81.5% to 85.8%; TransCLIP-ZS runs in 14.4 s vs. >150 min for transductive UPL on zero-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TransCLIP-ZS transduction to CLIP-ViT-B/16 zero-shot predictions | CLIP-ViT-B/16 · Average (11 datasets, zero-shot) | top-1 accuracy: 65.3 -> 70.3 | Helped |
| Add TransCLIP few-shot transduction with text regularization | CLIP-ViT-B/16 few-shot predictions · Average (11 datasets, 1-shot) | top-1 accuracy: 52.8 -> 73.2 | Helped |
| Add TransCLIP few-shot transduction with text regularization | CLIP-ViT-B/16 few-shot predictions · Average (11 datasets, 4-shot) | top-1 accuracy: 71.3 -> 75.9 | Helped |
| Add TransCLIP few-shot transduction with text regularization | CLIP-ViT-B/16 few-shot predictions · Average (11 datasets, 16-shot) | top-1 accuracy: 77.8 -> 78.4 | Helped |
| Add text regularization to TransCLIP-FS (ablation) | TransCLIP-FS · Average (11 datasets, 1-shot) | top-1 accuracy: 50.2 -> 73.2 | Helped |
| Add text regularization to TransCLIP-FS (ablation) | TransCLIP-FS · Average (11 datasets, 4-shot) | top-1 accuracy: 69.6 -> 75.9 | Helped |
| Add text regularization to TransCLIP-FS (ablation) | TransCLIP-FS · Average (11 datasets, 16-shot) | top-1 accuracy: 76.6 -> 78.4 | Helped |
| Update μ during TransCLIP optimization (ablation) | TransCLIP-ZS · ImageNet | top-1 accuracy: 69.7 -> 70.3 | Helped |
| Update Σ during TransCLIP optimization (ablation) | TransCLIP-ZS · ImageNet | top-1 accuracy: 68.7 -> 70.3 | Helped |
| Use Laplacian regularization (ablation) | TransCLIP-ZS · ImageNet | top-1 accuracy: 69.9 -> 70.3 | Helped |
| Set zero-shot text regularization λ to 0.1 instead of 1 (ablation) | TransCLIP-ZS · ImageNet | top-1 accuracy: 70.3 -> 56.3 | Hurt |
| Vary the number of Laplacian nearest neighbors from 3 to 5 or 10 (ablation) | TransCLIP-ZS · ImageNet | top-1 accuracy: 70.3; 70.3; 70.2 | No clear effect |
Surprising findings
- Transductive few-shot methods designed for vision-only backbones often underperform inductive zero-shot CLIP on VLMs; e.g., TF reaches 29.7% on ImageNet at 1-shot.
- Transductive gains shrink as the number of shots grows (e.g., at 16 shots the gain over few-shot methods is smaller).
Limitations
- Performance depends heavily on the quality of the text prompts; the KL text term can introduce biases toward frequent concepts in text embeddings.
- Gains over few-shot baselines diminish as the number of labeled shots increases (e.g., 16-shot).
Experimental setup
Datasets: ImageNet SUN397 FGVC Aircraft EuroSAT Stanford Cars Food101 Oxford Pets Flowers102 Caltech101 DTD UCF101 ImageNet-A
Benchmarks: ImageNet SUN397 FGVC Aircraft EuroSAT Stanford Cars Food101 Oxford Pets Flowers102 Caltech101 DTD UCF101 ImageNet-A
Metrics: top-1 accuracy runtime
Baselines: CLIP zero-shot CoOp Tip-Adapter-F PLOT TaskRes ProGrad TF BD-CSPN LaplacianShot PT-MAP TIM UPL
Builds on: CLIP Tip-Adapter TaskRes CoOp Unsupervised Prompt Learning
Closest papers by meaning
- A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.90)
- Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients (CVPR 2026 · similarity 0.89)
- TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.89)
- TGV-KV: Text-Grounded KV Eviction for Vision-Language Models (ICML 2026 · similarity 0.89)
- Frustratingly Easy Test-Time Adaptation of Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Realistic Test-Time Adaptation of Vision-Language Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.