Research map / Vision-language and multimodal / CLIP and zero-shot
Large Language Models are Good Prompt Learners for Low-Shot Image Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLaMP, which uses an LLaMA2-7B decoder as an adaptive prompt learner for the CLIP text encoder, via a KV-cache trick that trains only query/output projections and prompts, evaluated on base-to-novel and 16-shot classification over 11 datasets.
Problem
Low-shot image classification (zero-shot base-to-novel and few-shot) relies on CLIP text features from bare class names, which carry little class-specific information for fine-grained categories; directly using LLMs is hampered by the vision-language domain gap and data scarcity.
Approach
Treat an LLM as the prompt learner of the CLIP text encoder: learnable prompts are appended after cached LLM key/value states of a description query, only the query and output projections of the last LLM layer are tuned, and the resulting hidden states are projected into the CLIP text input space as class-specific prompts.
Key results
- Base-to-novel average harmonic mean of 81.27% vs 79.97% for PSRC (+1.30) and 71.70% for CLIP across 11 datasets.
- 16-shot average accuracy of 83.81% vs 82.87% for PSRC.
- On EuroSAT, novel accuracy improves by 9.76 points over PSRC (87.60 HM vs 82.32).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LLaMP with LLM-derived adaptive prompts instead of PSRC | CLIP ViT-B/16 low-shot classification · 11-dataset base-to-novel benchmark | average harmonic mean: +1.30 | Helped |
| Add parsed noun phrases from LLM responses to CLIP prompts without training (ablation) | CLIP · Average base-to-novel | harmonic mean: 71.70 -> 72.79 | Helped |
| Add LLM-based adaptive prompting to the ordinary prompting model with vision LoRA (ablation) | LLaMP without LLM · Average base-to-novel | harmonic mean: 79.22 -> 81.27 | Helped |
| Train LLM prompts and Query/Output projections rather than prompts alone (ablation) | LLaMP decoder · ImageNet base-to-novel | harmonic mean: 80.96 -> 81.27 | Helped |
| Use parsed noun phrases as textual priors instead of no textual priors (ablation) | LLaMP · ImageNet base-to-novel | harmonic mean: 81.08 -> 81.27 | Helped |
| Use the CLIP text encoder as a bridge instead of using the LLM alone as text encoder (ablation) | LLaMP · ImageNet base-to-novel | novel accuracy: 35.82 -> 77.71 | Helped |
| Use hybrid vision tuning (prompt ×6 + LoRA ×6) instead of prompt ×9 (ablation) | LLaMP vision encoder · ImageNet base-to-novel | harmonic mean: 80.81 -> 81.27 | Helped |
| Increase LLM prompt count from 2 to 16 (ablation) | LLaMP · ImageNet base-to-novel | harmonic mean: 80.94 -> 81.27 | Helped |
| Use LLaMP rather than PSRC | 16-shot image classification · FGVCAircraft | accuracy: +5.24 | Helped |
Surprising findings
- Using only an LLM as the text encoder (LLM-only) yields 35.82% novel accuracy, showing that the CLIP text encoder is essential as the bridge.
- Simply adding parsed noun phrases from LLM responses to ordinary CLIP prompts improves HM by about 1% without any training.
Limitations
- Vision and language only interact at the final feature level, limiting cross-modal benefit.
- Relies on an LLM-generated description per class, requiring LLM queries and noun-phrase parsing.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 UCF101 DTD EuroSAT
Metrics: base accuracy novel accuracy harmonic mean 16-shot accuracy
Baselines: CLIP CoOp CoCoOp KAPT ProDA MaPLe RPO PSRC
Builds on: CLIP CoOp CoCoOp PSRC MaPLe LoRA
Closest papers by meaning
- Making Large Vision Language Models to Be Good Few-Shot Learners (AAAI 2025 · similarity 0.91)
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.91)
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.90)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.90)
- ProAPO: Progressively Automatic Prompt Optimization for Visual Classification (CVPR 2025 · similarity 0.90)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.