Research map / Vision-language and multimodal / CLIP and zero-shot

Making Large Vision Language Models to Be Good Few-Shot Learners

AAAI 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Adapting VLMs for zero-shot classification · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Meta-learning instruction tuning of Qwen-VL-Chat with character-perturbation label augmentation and attribute-based candidate selection to make LVLMs strong 5-way 1-shot few-shot classifiers.

Problem

LVLMs struggle with few-shot classification: they fail to exploit support samples, favor specific response formats, and show strong positional bias toward early candidate options.

Approach

Meta-task instruction tuning (learn-to-learn) of an LVLM on N-way K-shot formatted datasets, combined with label augmentation via character perturbation of class names to force reliance on support data, and an inference-time candidate selection that uses aggregated attribute-description similarity to narrow candidates and re-query the model.

Key results

• Full method reaches 97.45% average accuracy over eight datasets in 5-way 1-shot, versus 43.12% for untuned Qwen-VL and 96.39% with meta-learning alone.
• Ours achieves 98.24 MINI, 95.02 CIFAR, 98.06 TIERED, 96.40 CUB, outperforming prior SOTA such as PTMAP-SF-SOT (90.44 average on four general datasets).
• On fine-grained datasets, average accuracy of 97.60% vs. 84.56% for SRM; candidate selection alone raises training-free Qwen-VL by 25.69 points on average.

Measured effects reported in the paper

Surprising findings

• Untuned Qwen-VL shows severe positional bias, concentrating answers in early candidate positions, and performs poorly even with support examples.
• Attribute-description integration directly into the prompt degraded performance, so similarity-based candidate selection was used instead.

Limitations

• Relies on the base-to-novel class split of ELEVATER datasets and evaluation on standard FSL benchmarks; generalization beyond these is not shown.
• Single attributes alone are weak; performance on CUB lags CIFAR, suggesting fine-grained attribute descriptions are needed.
• Candidate selection requires an extra re-query step when initial inference falls outside candidates, adding inference cost.

Experimental setup

Datasets: ELEVATER (13 datasets, instruction tuning) MiniImageNet CIFAR-FS TieredImageNet CUB Stanford Dogs FGVC-Aircraft Oxford 102 Flower Stanford Cars

Benchmarks: MiniImageNet CIFAR-FS TieredImageNet CUB Stanford Dogs FGVC-Aircraft Oxford 102 Flower Stanford Cars

Metrics: Acc Acc_occur Acc_CLIP normalized standard deviation of answer positions

Baselines: BAVARDAGE EASY PEMnE-BMS PTMAP-SF-SOT P>M>F CAML FRN TDM MCL-Katz DeepBDC LCCRN SRM

Builds on: Qwen-VL-Chat Q-LoRA ELEVATER SBERT

Closest papers by meaning

• Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.92)
• Large Language Models are Good Prompt Learners for Low-Shot Image Classification (CVPR 2024 · similarity 0.91)
• Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning (CVPR 2025 · similarity 0.90)
• A Closer Look at the Few-Shot Adaptation of Large Vision-Language Models (CVPR 2024 · similarity 0.89)
• Pushing the Limit of Fine-Tuning for Few-Shot Learning: Where Feature Reusing Meets Cross-Scale Attention (AAAI 2024 · similarity 0.89)
• Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.89)