Research map / Label-efficient and robust learning / Self-supervised and contrastive

ImagineFSL: Self-Supervised Pretraining Matters on Imagined Base Set for VLM-based Few-shot Learning

CVPR 2025 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Pretext-task visual pre-training · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Pretrains a CLIP adapter with a self-supervised method (HoM-DINO) on a purely synthetic ImageNet-based base set generated by GPT-4/Llama/SD3, then fine-tunes for few-shot recognition.

Problem

Few-shot adaptation of CLIP suffers from data scarcity; existing methods use synthetic images only as complements to real few-shot images rather than as a standalone pretraining source.

Approach

Treat synthetic images as an independent imagined base set (iBase) for self-supervised pretraining of a CLIP adapter; the HoM-DINO method models patch-token distributions via higher-order moments plus masked image modeling, with SyntAug (two T2I samples from the same caption) as view construction, and captions are generated with CoT and ICL.

Key results

• On 11-dataset 1-shot to 16-shot recognition, ImagineFSL reaches 76.1% at 1-shot and 84.4% at 16-shot average accuracy, and ImagineFSLLoRA reaches 82.1% average over 1/2/4/8/16-shot settings in Table 2, outperforming CLIP-LoRA (77.7%) and GalLoP (79.0%).
• On domain generalization with 16 shots, ImagineFSLLoRA averages 63.2% vs. 61.9% for CoCoLe, with new state of the art on ImageNet-Sketch (53.8%) and ImageNet-R (80.0%).
• On zero-shot recognition over 11 datasets, ImagineFSLLoRA averages 73.0% vs. 70.5% for DMN; HoM-DINO pretraining beats DINOv2 by 2.1%/0.7% in 1-shot/16-shot accuracy.

Measured effects reported in the paper

Surprising findings

• Self-supervised pretraining on synthetic images outperforms supervised pretraining on the same data (HoM-DINO beats SL by 2.3%/1.1% in 1-shot/16-shot).
• MAE underperforms markedly as a self-supervised pretraining method for few-shot CLIP adaptation, attributed to pixel-space reconstruction.

Limitations

• Pretraining overhead (about 9 hours on 4 RTX 4090 GPUs for ViT-B/16)
• Underutilization of synthetic captions
• Frozen encoder may not effectively exploit expanded data (saturation from 0.3M to 0.45M images)

Experimental setup

Datasets: ImageNet-1K ImageNet-21K SyntIN1K SyntIN1.5K Caltech101 FGVC Aircraft Stanford Cars Food-101 Oxford Pets Flowers102 DTD EuroSAT

Benchmarks: ImageNet Caltech101 FGVC Aircraft Stanford Cars Food-101 Oxford Pets Flowers102 DTD EuroSAT SUN397 UCF101 ImageNet-V2

Metrics: average accuracy FID diversity (std of DINOv2 features) latency memory parameter ratio

Baselines: IsSynth CaFo DISEF DataDream CoOp CoCoOp MaPLe PromptSRC Tip-Adapter(f) CLIP-LoRA LLaMP GalLoP

Builds on: DINO CLIP Tip-Adapter DataDream CaFo DINOv2

Closest papers by meaning

• Scaling Laws of Synthetic Images for Model Training ... for Now (CVPR 2024 · similarity 0.88)
• Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.87)
• Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.87)
• Provably Improving Generalization of Few-shot models with Synthetic Data (ICML 2025 · similarity 0.87)
• Learning Vision from Models Rivals Learning Vision from Data (CVPR 2024 · similarity 0.87)
• Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP (ICLR 2024 · similarity 0.87)