Research map / Vision-language and multimodal / CLIP and zero-shot
Lipsum-FT: Robust Fine-Tuning of Zero-Shot Models Using Random Text Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Lipsum-FT, a robust fine-tuning method for CLIP that regularizes fine-tuned vision features toward the zero-shot model using logits computed against random text token embeddings, motivated by an energy-based view of vision-language models.
Problem
Standard fine-tuning of zero-shot vision-language models (CLIP) improves reference-data accuracy but degrades robustness under distribution shifts, and the feature distortion theory does not adequately explain this trade-off for CLIP-ViT.
Approach
Interpret CLIP as a joint energy-based model and identify an energy gap between fine-tuned and zero-shot models as the cause of robustness loss; regularize fine-tuning with an MSE logit-matching loss on randomly generated text tokens (sampled from the vocabulary), minimizing the energy gap across all potential discriminative heads.
Key results
- On DomainNet with CLIP ViT-B/16, Lipsum-FT reaches 89.0 on R and 56.2 on S, versus 88.5 and 50.6 for standard FT.
- On ImageNet with ViT-B/16, Lipsum-FT achieves 75.9 on ImageNet-R and 51.4 on ImageNet-Sketch, versus 68.5 and 48.0 for FT.
- Lipsum-FT attains lower ECE and NLL than the baselines on distribution shift data in most settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace KLD with MSE loss (ablation) | CAR-FT · DomainNet-P | accuracy: 64.4 -> 65.7 | Helped |
| use random rather than fixed text guidance (ablation) | CAR-FT_MSE · DomainNet-P | accuracy: 65.7 -> 66.3 | Helped |
| combine MSE loss with random text guidance (ablation) | CAR-FT · DomainNet-P | accuracy: 64.4 -> 66.3 | Helped |
| fine-tune CLIP with Lipsum-FT rather than standard FT | CLIP ViT-B/16 · ImageNet-R | accuracy: 68.5 -> 75.9 | Helped |
| fine-tune CLIP with Lipsum-FT rather than standard FT | CLIP ViT-B/16 · DomainNet-I | accuracy: 41.2 -> 46.0 | Helped |
Surprising findings
- Feature distortion on distribution-shift data is larger than on reference data for CLIP-ViT fine-tuning, contradicting the feature distortion theory.
- LP-FT does not improve distribution shift robustness over standard FT with zero-shot head initialization in CLIP-ViT fine-tuning.
Limitations
- Text guidance is random rather than task-specific; the authors note designing a tailored text guidance strategy as future work.
- TPGM and Soup/Ensemble hyperparameter selection used test or validation splits of the reference data, which the authors acknowledge complicates test-set interpretation.
Experimental setup
Datasets: DomainNet-R ImageNet
Benchmarks: DomainNet-P DomainNet-C DomainNet-I DomainNet-S ImageNet-V2 ImageNet-R ImageNet-A ImageNet-Sketch
Metrics: accuracy expected calibration error negative log-likelihood
Baselines: FT EMA L2SP KD CAR-FT WiSE TPGM Model soup LP-FT
Builds on: CLIP WiSE-FT CAR-FT LP-FT Joint Energy-based Models
Closest papers by meaning
- Towards Calibrated Robust Fine-Tuning of Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustness (CVPR 2024 · similarity 0.87)
- Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.87)
- SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes (NeurIPS 2025 · similarity 0.86)
- Adaptive Parameter Selection for Tuning Vision-Language Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.