Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning

Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning

CVPR 2025 · Label-efficient and robust learning › Zero-/few-shot and meta-learning · approach: Transductive and semantic-dictionary zero-shot · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SWAT, a two-stage retrieval-augmented finetuning method for few-shot recognition that finetunes a CLIP visual encoder on mixed LAION-retrieved and few-shot data, then retrains the classifier on few-shot data only.

Problem

Few-shot recognition with a pretrained vision-language model, where retrieved pretraining data (retrieval-augmented learning) suffers from imbalanced class distribution and domain gaps relative to the few-shot examples, and prior FSR methods use unrealistically large validation sets.

Approach

Extends retrieval-augmented learning from zero-shot to few-shot recognition: full finetuning of the visual encoder on mixed retrieved and few-shot data (with CutMix) in stage one, followed by classifier retraining on few-shot data only in stage two, mirroring decoupled long-tail learning and transfer learning to address imbalance and domain gaps.

Key results

• SWAT achieves 73.5% / 76.0% / 78.2% mean accuracy at 4/8/16 shots, outperforming CLAP by 6.6, 6.0 and 5.3 points.
• Few-shot finetuning of the visual encoder alone (FSFT) reaches 69.7/73.3/76.3%, already beating CLAP by about 3 points.
• Finetuning on retrieved data only underperforms: REAL-Linear reaches 64.8% and retrieved-only stage-1 with classifier retraining reaches 66.5% at 16 shots, below the few-shot-only finetuning.

Measured effects reported in the paper

Surprising findings

• Finetuning the visual encoder solely on few-shot data already outperforms prior few-shot methods, contrary to the PEFT-focused literature.
• Finetuning on large retrieved data underperforms state-of-the-art zero-shot methods due to imbalance and domain gap, and a binary classifier distinguishes retrieved from few-shot images with over 90% accuracy.

Limitations

• Hyperparameters are taken from literature defaults and not tuned per task, so they may be suboptimal.
• Retrieval relies on concept names via string matching, which may be limited for rich descriptions.
• Not evaluated in a real-world data annotation application; the domain-gap and imbalance factors are not decoupled.

Experimental setup

Datasets: Semi-Aves Flowers102 FGVC-Aircraft EuroSAT DTD OxfordPets Food101 StanfordCars ImageNet LAION-400M

Benchmarks: Semi-Aves Flowers102 FGVC-Aircraft EuroSAT DTD OxfordPets Food101 StanfordCars ImageNet

Metrics: mean accuracy over nine datasets

Baselines: OpenCLIP REAL-Prompt REAL-Linear CoOp PLOT CLIP-Adapter TIP-Adapter TaskRes CrossModal-LP CLAP

Builds on: CLIP REAL (retrieval-augmented learning for zero-shot recognition) CLAP CutMix decoupling learning for long-tailed recognition

Closest papers by meaning

• Making Large Vision Language Models to Be Good Few-Shot Learners (AAAI 2025 · similarity 0.90)
• Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.89)
• Pushing the Limit of Fine-Tuning for Few-Shot Learning: Where Feature Reusing Meets Cross-Scale Attention (AAAI 2024 · similarity 0.88)
• Revisiting Few-Shot Object Detection with Vision-Language Models (NeurIPS 2024 · similarity 0.88)
• BECLR: Batch Enhanced Contrastive Few-Shot Learning (ICLR 2024 · similarity 0.88)
• Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.88)