Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Logits DeConfusion with CLIP for Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Logits DeConfusion (LDC), which combines a multi-level adapter fusion module and a residual inter-class deconfusion module to learn and remove CLIP's inter-class logit confusion for few-shot classification.
Problem
CLIP's zero-shot logits show serious inter-class confusion in downstream tasks, which hurts accuracy in few-shot learning adaptation.
Approach
Model inter-class confusion as an additive noise term on CLIP logits, learned by adapters that take both zero-shot logits and visual features as priors, and subtract it through a residual structure; fuse with an MLP-based logits branch via adaptive weighting, with an L1 similarity loss to prevent over-deconfusion.
Key results
- Achieves 66.63% on ImageNet 16-shot with ResNet-50, 0.24 above the previous best (FAR, 66.39%).
- Averages 79.78% across 11 datasets at 16 shots, versus 76.18% for Proto-CLIP-F.
- Ablation: ICD module raises 11-dataset average by 19.07% over the ResNet-50 baseline (58.87 to 77.94).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LDC rather than FAR | CLIP on ImageNet, 16-shot · ImageNet, 16-shot | accuracy: 66.39 -> 66.63 | Helped |
| Use LDC rather than Proto-CLIP-F | CLIP on the 11-dataset average, 16-shot · 11-dataset average, 16-shot | accuracy: 76.18 -> 79.78 | Helped |
| Apply LDC on ImageNet-Sketch | CLIP, ResNet-50, 16-shot · ImageNet-Sketch | accuracy: 36.69 -> 35.52 | Hurt |
| Add MAF to the ZS-CLIP baseline (ablation) | CLIP, ResNet-50 · 11-dataset average, 16-shot | accuracy: 58.87 -> 70.83 | Helped |
| Add ICD to the ZS-CLIP baseline (ablation) | CLIP, ResNet-50 · 11-dataset average, 16-shot | accuracy: 58.87 -> 77.94 | Helped |
| Add ALF to the MAF + ICD method (ablation) | LDC, ResNet-50 · 11-dataset average, 16-shot | accuracy: 78.82 -> 79.78 | Helped |
| Add more feature levels to MAF (ablation) | LDC, ResNet-50 · 11-dataset average, 16-shot | accuracy: 75.95 -> 77.48 -> 79.10 -> 79.78 | Helped |
| Use weighted fusion with weights 0.1/0.2/0.3/0.4 instead of equal weights (ablation) | MAF · 11-dataset average, 16-shot | accuracy: 79.71 -> 79.78 | Helped |
| Add the projector to MAF (ablation) | LDC, ResNet-50 · 11-dataset average, 16-shot | accuracy: 78.94 -> 79.78 | Helped |
| Add the residual structure to ICD (ablation) | LDC, ResNet-50 · 11-dataset average, 16-shot | accuracy: 78.54 -> 79.78 | Helped |
| Use adaptive α instead of fixed α=0.5 for logits fusion (ablation) | ALF · 11-dataset average, 16-shot | accuracy: 79.24 -> 79.78 | Helped |
| Add both similarity losses, LMAF Sim and LICD Sim (ablation) | LDC training objective · 11-dataset average, 16-shot | accuracy: 79.03 -> 79.78 | Helped |
Surprising findings
- Performance on ImageNet-Sketch under domain shift is below several baselines (35.52 vs 36.69 for FAR), which the authors attribute to limited suitability for large domain gaps.
Limitations
- Underperforms on ImageNet-Sketch, indicating weakness under large domain differences.
- Gains at 1-shot on ImageNet are negative relative to some baselines (e.g., 60.48 vs 62.04 for APE).
Experimental setup
Datasets: ImageNet Caltech101 DTD EuroSAT FGVCAircraft Flowers102 Food101 OxfordPets StanfordCars SUN397 UCF101
Benchmarks: ImageNet Caltech101 DTD EuroSAT FGVCAircraft Flowers102 Food101 OxfordPets StanfordCars SUN397 UCF101 ImageNet-V2
Metrics: accuracy
Baselines: CoOp VT-CLIP Tip-Adapter Tip-Adapter-F SuS-X FAR CALIP-FS SGVA-CLIP Proto-CLIP-F APE CLIP-Adapter DAC-V
Builds on: CLIP Tip-Adapter CoOp CLIP-Adapter
Closest papers by meaning
- AMU-Tuning: Effective Logit Bias for CLIP-based Few-shot Learning (CVPR 2024 · similarity 0.85)
- Conformal Prediction for Zero-Shot Models (CVPR 2025 · similarity 0.83)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.82)
- VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions (NeurIPS 2025 · similarity 0.82)
- Scaled Decoupled Distillation (CVPR 2024 · similarity 0.82)
- CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.