Research map / Vision-language and multimodal / CLIP and zero-shot
Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeMul, a description-free multi-prompt learning method that distills learnable CLIP prompts into the GPT text-embedding space and uses learnable prompt weights for few-shot image classification.
Problem
Description-based LLM prompting for VLMs suffers from high variability and low reliability of generated descriptions, and existing learnable prompt methods do not account for the varying importance of multiple prompts.
Approach
Instead of extracting text descriptions from an LLM, map learnable CLIP prompt embeddings into the GPT embedding space via a learned mapping (with a cyclic mapping loss) and distill semantics from GPT class embeddings; add learnable per-prompt weights with L1 sparsity to weight multiple prompts per class.
Key results
- DeMul reaches 75.5% average top-1 in 1-shot and 85.3% in 16-shot across 11 datasets, versus GalLoP at 73.7% and 84.5%.
- Compared with zero-shot CLIP, DeMul improves average accuracy by 10.5% at 1-shot and 20.3% at 16-shot (as stated by the authors).
- Using text-embedding-3-large as the distillation target gives the best 16-shot average (85.3%) over text-embedding-3-small (83.4%) and ada-002 (82.5%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DeMul instead of GalLoP | CLIP ViT-B/16 few-shot classification · 11-dataset average, 1-shot | Top-1 accuracy: 73.7 -> 75.5 | Helped |
| Use DeMul instead of GalLoP | CLIP ViT-B/16 few-shot classification · 11-dataset average, 16-shot | Top-1 accuracy: 84.4 -> 85.3 | Helped |
| Add LLM distillation loss (ablation) | DeMul · 11-dataset average, 1-shot | Top-1 accuracy: 75.0 -> 75.5 | Helped |
| Add LLM distillation loss (ablation) | DeMul · 11-dataset average, 2-shot | Top-1 accuracy: 78.0 -> 78.6 | Helped |
| Add LLM distillation loss (ablation) | DeMul · 11-dataset average, 4-shot | Top-1 accuracy: 80.4 -> 80.9 | Helped |
| Add LLM distillation loss (ablation) | DeMul · 11-dataset average, 8-shot | Top-1 accuracy: 83.0 -> 83.6 | Helped |
| Add LLM distillation loss (ablation) | DeMul · 11-dataset average, 16-shot | Top-1 accuracy: 84.8 -> 85.3 | Helped |
| Add prompt weighting (ablation) | DeMul · 11-dataset average, 1-shot | Top-1 accuracy: 75.2 -> 75.5 | Helped |
| Add prompt weighting (ablation) | DeMul · 11-dataset average, 2-shot | Top-1 accuracy: 77.8 -> 78.6 | Helped |
| Add prompt weighting (ablation) | DeMul · 11-dataset average, 4-shot | Top-1 accuracy: 80.1 -> 80.9 | Helped |
| Add prompt weighting (ablation) | DeMul · 11-dataset average, 8-shot | Top-1 accuracy: 83.1 -> 83.6 | Helped |
| Add prompt weighting (ablation) | DeMul · 11-dataset average, 16-shot | Top-1 accuracy: 85.2 -> 85.3 | Helped |
Surprising findings
- Differences in the ablation (removing distillation or weighting) were small, which the authors acknowledge as not substantial.
Limitations
- Performance still depends on the distribution of the few-shot training images.
- Learnable prompt vectors are shared across all classes, which hinders class-specific prompts.
- Relies on a proprietary GPT embedding API, so reproducibility depends on access.
Experimental setup
Datasets: ImageNet SUN397 Stanford Cars UCF101 Caltech101 EuroSAT FGVC Aircraft Food101 DTD Oxford Flowers Oxford Pets WordNet
Benchmarks: ImageNet SUN397 Stanford Cars UCF101 Caltech101 EuroSAT FGVC Aircraft Food101 DTD Oxford Flowers Oxford Pets
Metrics: top-1 accuracy
Baselines: Zero-Shot CLIP dCLIP WaffleCLIP Linear Probe CLIP CoOp MaPLe PromptSRC GalLoP
Builds on: CoOp CLIP ProDA dCLIP GalLoP
Closest papers by meaning
- Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.92)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.91)
- ProAPO: Progressively Automatic Prompt Optimization for Visual Classification (CVPR 2025 · similarity 0.91)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.91)
- Large Language Models are Good Prompt Learners for Low-Shot Image Classification (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.