Research map / Vision-language and multimodal / CLIP and zero-shot

Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIP

AAAI 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: CLIP training recipes and data curation · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TIMO, a training-free CLIP few-shot method with text-guided-image and image-guided-text components for mutual image-text guidance, plus TIMO-S with tuned hyperparameters that beats training-required methods.

Problem

Training-free CLIP few-shot classification models image and text modalities independently, causing anomalous image-image matches and varying quality of LLM-generated text prompts.

Approach

Mutual guidance between modalities: text prototypes reweight/select image support features (TGI, instance-based transfer), while image prototypes guide a closed-form softmax-weighted combination of text prompts (IGT), with both plugged into existing methods like GDA-CLIP and Tip-Adapter.

Key results

• TIMO-S averages 71.63% over 11 datasets at 16 shots-style table, outperforming the best training-free method GDA-CLIP (69.87% avg) and surpassing training-required Cross-Modal Partial (71.30%) by 0.33%.
• On ImageNet 16-shot, TIMO-S takes 6.0 s versus 787.5 s for APE and 257.4 s for Tip-Adapter, roughly 100x less time than training-required methods.
• Plugging TGI and IGT into GDA-CLIP raises its 11-dataset average from 69.87% to 71.19% (TIMO).

Measured effects reported in the paper

Surprising findings

• Fine-tuning TIMO with TaskRes (TIMO-F) generally decreased performance, with average 71.11% vs 71.19% without fine-tuning.
• The mutual guidance gains appear to come at the cost of diversity among individual classifiers, per a Q-statistic analysis.

Limitations

• Gains over prior training-free methods are modest (around 1-1.3% average), and TIMO-S relies on a validation-set grid search over beta and gamma.
• Fine-tuning does not help consistently, so the method does not exploit training to unlock more CLIP potential.
• Ensemble analysis suggests the error reduction comes at the cost of reduced diversity among individual classifiers.

Experimental setup

Datasets: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet-V2

Benchmarks: ImageNet Caltech101 OxfordPets StanfordCars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet-V2

Metrics: top-1 accuracy average accuracy time cost

Baselines: Zero-Shot CLIP Tip-Adapter APE GDA-CLIP Tip-Adapter-F Cross-Modal Partial APE-T LP++ CLAP Linear Probe CLIP CoOp

Builds on: CLIP Tip-Adapter GDA-CLIP CuPL APE

Closest papers by meaning

• A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.90)
• Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
• Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.88)
• Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.88)
• Learning to Prompt with Text Only Supervision for Vision-Language Models (AAAI 2025 · similarity 0.88)
• Compound Text-Guided Prompt Tuning via Image-Adaptive Cues (AAAI 2024 · similarity 0.88)