Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Self-Prompt Mechanism for Few-Shot Image Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a self-prompt mechanism (SPM) that derives prompts from the ViT's own deep-layer features via spatial and channel selection, injected into self-attention queries, and tunes only these prompt parameters (≤2%) for few-shot recognition.
Problem
Few-shot image recognition with few labeled examples; existing text-prompt methods (CLIP-based) suffer from modality gap between text prompts and image features, extra text-encoder cost, and noisy text descriptions.
Approach
Generate prompts intrinsically from the image encoder's deep-layer features via learnable spatial mask and channel matrix selection, feed them back into the last three transformer blocks' query computation (top-down), and fine-tune only these prompt parameters plus biases at meta-test time, keeping encoder weights frozen.
Key results
- Achieves 93.7% 5-way 1-shot and 98.3% 5-way 5-shot accuracy on Mini-ImageNet, versus 93.1/98.0 for PMF (ViT-small).
- Reaches 82.4% 1-shot on CIFAR-FS, above SP-CLIP* at 81.9% and PMF at 81.1%.
- On cross-domain EuroSAT and CropDiseases 5-way 1-shot, SPM reaches 74.97% and 84.43%, versus 70.74% and 80.79% for PMF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add spatial selection (ablation) | SPM · Mini-ImageNet, 5-way 1-shot | accuracy: 93.35 -> 93.72 | Helped |
| add spatial selection (ablation) | SPM · CIFAR-FS, 5-way 1-shot | accuracy: 82.02 -> 82.42 | Helped |
| add spatial selection (ablation) | SPM · EuroSAT, 5-way 1-shot | accuracy: 73.87 -> 74.97 | Helped |
| add spatial selection (ablation) | SPM · CropDisease, 5-way 1-shot | accuracy: 83.66 -> 84.43 | Helped |
| add channel interaction (ablation) | SPM · Mini-ImageNet, 5-way 1-shot | accuracy: 93.26 -> 93.72 | Helped |
| add channel interaction (ablation) | SPM · CIFAR-FS, 5-way 1-shot | accuracy: 81.95 -> 82.42 | Helped |
| add channel interaction (ablation) | SPM · EuroSAT, 5-way 1-shot | accuracy: 71.31 -> 74.97 | Helped |
| add channel interaction (ablation) | SPM · CropDisease, 5-way 1-shot | accuracy: 81.39 -> 84.43 | Helped |
| use identity projection instead of MLP projection (ablation) | SPM · Mini-ImageNet, 5-way 1-shot | accuracy: 93.25 -> 93.72 | Helped |
| use identity projection instead of MLP projection (ablation) | SPM · CIFAR-FS, 5-way 1-shot | accuracy: 81.22 -> 82.42 | Helped |
| apply self-prompt mechanism to deeper Transformer layers (ablation) | SPM · single-domain and cross-domain few-shot datasets | accuracy | Helped |
| use SPM instead of PMF with ViT-small | few-shot recognition · EuroSAT and CropDisease, cross-domain | accuracy: 70.74 -> 74.97; 80.79 -> 84.43 | Helped |
Surprising findings
- Identity projection of self-prompts outperformed MLP and linear projections at meta-test time, even though MLP gave higher meta-training accuracy.
- Applying self-prompts to deeper layers worked better than shallow layers, which the authors attribute to high-level semantic information.
Limitations
- Gains over prior state of the art are modest (about 0.5 to 1.9 points on several benchmarks).
- Cross-domain gains on medical datasets (ChestX, ISIC) are marginal (0.56% and 0.59%).
Experimental setup
Datasets: Mini-ImageNet CIFAR-FS CropDiseases EuroSAT ISIC ChestX
Benchmarks: Mini-ImageNet CIFAR-FS CropDiseases EuroSAT ISIC ChestX
Metrics: accuracy
Baselines: SP-CLIP PMF SUN SCR GEL PLCM DeepEMD StyleAdv-FT STARTUP FT-All
Builds on: Vision Transformer prototypical networks PMF visual prompt tuning
Closest papers by meaning
- SEC-Prompt:SEmantic Complementary Prompting for Few-Shot Class-Incremental Learning (CVPR 2025 · similarity 0.87)
- Visual and Semantic Prompt Collaboration for Generalized Zero-Shot Learning (CVPR 2025 · similarity 0.87)
- Multi-Label Few-Shot Image Classification via Pairwise Feature Augmentation and Flexible Prompt Learning (AAAI 2025 · similarity 0.87)
- PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR 2024 · similarity 0.86)
- Large Language Models are Good Prompt Learners for Low-Shot Image Classification (CVPR 2024 · similarity 0.86)
- Self-cross Feature based Spiking Neural Networks for Efficient Few-shot Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.