Research map / Vision-language and multimodal / CLIP and zero-shot
AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes CLIP-based few-shot learning through a logit-bias formulation and proposes AMU-Tuning, which trains a feature-initialized linear predictor on MoCov3 auxiliary features with multi-branch loss and fuses it via a kurtosis-based uncertainty weight.
Problem
CLIP-based few-shot classification methods are hard to compare and their key factors are poorly understood, limiting exploration of CLIP's few-shot potential.
Approach
Unifies prior CLIP few-shot methods as zero-shot CLIP logits plus a learned logit bias, disassembles the bias into features, predictor, and fusion, and proposes complementary auxiliary features (MoCov3), a feature-initialized linear probe trained with a multi-branch loss, and an uncertainty-based (kurtosis) fusion weight.
Key results
- On ImageNet-1K with RN50 and 16 shots, AMU-Tuning reaches 70.02% vs. 68.79% for CaFo and 65.51% for Tip-Adapter-F.
- AMU-Tuning outperforms CaFo* (CaFo without DALL-E and GPT-3) at all shot settings on ImageNet-1K, by 1.02 to 1.97 points.
- With ViT-B/16 on ImageNet-1K 16-shot, AMU-Tuning reaches 74.98% vs. 74.48% for CaFo; on OOD ImageNet-A it reaches 52.05% with ViT backbone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AMU-Tuning instead of CaFo | CLIP RN50 few-shot classification · ImageNet-1K, 4-shot | accuracy: 65.64 -> 65.92 | Helped |
| Use AMU-Tuning instead of CaFo | CLIP RN50 few-shot classification · ImageNet-1K, 8-shot | accuracy: 66.86 -> 68.25 | Helped |
| Use AMU-Tuning instead of CaFo | CLIP RN50 few-shot classification · ImageNet-1K, 16-shot | accuracy: 68.79 -> 70.02 | Helped |
| Use AMU-Tuning instead of CaFo | CLIP RN50 few-shot classification · ImageNet-1K, 1-shot | accuracy: 63.80 -> 62.60 | Hurt |
| Use AMU-Tuning instead of CaFo | CLIP RN50 few-shot classification · ImageNet-1K, 2-shot | accuracy: 64.34 -> 64.25 | Hurt |
| Use AMU-Tuning instead of MaPLe | CLIP ViT few-shot model transferred without retraining · ImageNet-A | accuracy: 50.90 -> 52.05 | Helped |
| Add auxiliary features (ablation) | AMU-Tuning on ImageNet-1K · ImageNet-1K, 16-shot | accuracy: 65.34 -> 69.35 | Helped |
| Add multi-branch training of feature-initialized LP (ablation) | AMU-Tuning with auxiliary features · ImageNet-1K, 16-shot | accuracy: 69.35 -> 69.72 | Helped |
| Add uncertainty-based fusion (ablation) | AMU-Tuning with auxiliary features and MTFi · ImageNet-1K, 16-shot | accuracy: 69.72 -> 70.02 | Helped |
| Use DINO instead of CLIP auxiliary features (ablation) | Auxiliary-feature logit prediction · ImageNet-1K, 16-shot | fusion accuracy: 65.34 -> 68.32 | Helped |
| Use MoCov3 instead of DINO auxiliary features (ablation) | Auxiliary-feature logit prediction · ImageNet-1K, 16-shot | fusion accuracy: 68.32 -> 69.35 | Helped |
| Train the bias branch individually instead of jointly with zero-shot CLIP (ablation) | LP bias branch with CLIP auxiliary features · ImageNet-1K, 16-shot | accuracy: 11.23 -> 56.93 | Helped |
Surprising findings
- Complementarity of auxiliary features to zero-shot CLIP matters more than their individual superiority for the final fused accuracy (e.g., DINO fuses better than CLIP despite lower standalone accuracy).
- Joint training of the bias branch with zero-shot CLIP performs far worse than training the bias branch individually (e.g., 11.23% vs 56.93% for CLIP features).
Limitations
- Few-shot gains over CaFo are absent at 1-shot and 2-shot settings because CaFo uses DALL-E generated extra samples.
- Hyperparameters (lambda, rho) are chosen by cross-validation, and fusion hyperparameters require validation data.
- Auxiliary feature selection relies on a per-dataset superiority and complementarity analysis.
Experimental setup
Datasets: ImageNet-1K StanfordCars Caltech101 UCF101 Flowers102 Food101 DTD EuroSAT FGVCAircraft OxfordPets SUN397 ImageNet-V2
Benchmarks: ImageNet-1K StanfordCars Caltech101 UCF101 Flowers102 Food101 DTD EuroSAT FGVCAircraft OxfordPets SUN397 ImageNet-V2
Metrics: top-1 accuracy (%)
Baselines: Zero-shot CLIP LP-CLIP CoOp CLIP-Adapter VT-CLIP Tip-Adapter-F CaFo CaFo* CoCoOp MaPLe
Builds on: CLIP Tip-Adapter CaFo CLIP-Adapter CoOp
Closest papers by meaning
- Pushing the Limit of Fine-Tuning for Few-Shot Learning: Where Feature Reusing Meets Cross-Scale Attention (AAAI 2024 · similarity 0.87)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.87)
- Making Large Vision Language Models to Be Good Few-Shot Learners (AAAI 2025 · similarity 0.86)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.86)
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.86)
- Transductive Zero-Shot and Few-Shot CLIP (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.