Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Finetune Like You Pretrain: Improved Finetuning of Zero-Shot Vision Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use contrastive finetuning instead of cross-entropy finetuning | CLIP ViT-B/16 · ImageNet ID and OOD | accuracy: +1.2% ID and +5.4% OOD | Helped |
| Use contrastive finetuning instead of LP-FT | CLIP ViT-B/16 · ImageNet ID | accuracy: +1.1% ID | Helped |
| Use contrastive finetuning instead of LP-FT | CLIP ViT-B/16 · WILDS-iWILDCam ID and OOD | macro F1: +2.5% ID and +1% OOD | Helped |
| Replace FLYP's contrastive objective with cross-entropy (ablation) | FLYP · WILDS-iWILDCam | ID and OOD performance: 2% worse ID and 0.6% worse OOD | Hurt |
| Freeze the language encoder during contrastive finetuning (ablation) | FLYP · WILDS-iWILDCam | ID and OOD performance: drop of 2.5% ID and 1.3% OOD | Hurt |
| Mask same-class collisions in the contrastive loss (ablation) | FLYP · PatchCamelyon | accuracy: 1.3% worse | Hurt |
| Use one prompt template instead of 80 (ablation) | FLYP · ImageNet ID and OOD | accuracy | No clear effect |
| Add cross-entropy loss to the contrastive FLYP objective (ablation) | FLYP · WILDS-iWILDCam | ID performance: drop of 2.64% | Hurt |
| Add cross-entropy loss to the contrastive FLYP objective (ablation) | FLYP · WILDS-iWILDCam | OOD performance: drop of 0.5% | Hurt |
| Reduce ImageNet batch size from 512 to 256 (ablation) | FLYP · ImageNet | OOD accuracy: drop of 1% | Hurt |
| Reduce FMoW batch size by half (ablation) | FLYP · WILDS-FMoW | OOD accuracy: drop in OOD accuracy of 0.5% | Hurt |
| Use 32-shot FLYP rather than LP-FT | CLIP ViT-B/16 · Rendered SST2 | accuracy: 4.4% | Helped |
Closest papers by meaning
- Anchor-based Robust Finetuning of Vision-Language Models (CVPR 2024 · similarity 0.87)
- Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization (ICLR 2024 · similarity 0.86)
- Retrieval-Enhanced Contrastive Vision-Text Models (ICLR 2024 · similarity 0.86)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.86)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.86)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.