Research map / Vision-language and multimodal / CLIP and zero-shot

Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification

CVPR 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Self-distillation for vision-language pretraining · 29 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Distills a black-box CLIP teacher into an ImageNet-pretrained vision student via align-distill-predict stages, improving OOD domain generalization on DG benchmarks.

Problem

Distilling a vision-language model into a small unimodal image classifier with limited labeled task data often improves in-domain accuracy but fails to transfer the VLM's out-of-distribution generalization, especially when the teacher is a black box.

Approach

Align the pretrained student's features to the CLIP image and text embeddings via a frozen-backbone linear projection head first, then distill the backbone with the same loss, and finally use CLIP text embeddings as the classifier head; a white-box self-distillation variant (VL2V-SD) distills text-encoder invariances into the CLIP image encoder.

Key results

• VL2V-ADiP with ViT-B/16 ImageNet-initialized student reaches 75.48% average OOD accuracy versus 73.97% for the best baseline RISE (S), about 1.5 points gain.
• VL2V-SD in the white-box setting reaches 77.73% average OOD accuracy versus 76.66% for MIRO (S) and 70.60% for CLIP zero-shot.
• Distillation with ViT-S/16, DeiT-S/16 and ResNet-50 students improves average OOD accuracy over KD (S) in all cases, e.g. 71.73% vs 69.08% for ViT-S/16.

Measured effects reported in the paper

Surprising findings

• Combining the align and distill stages into one joint training step drops OOD accuracy by 6.3 points (79.50 to 73.24), showing alignment before distillation matters.
• CLIP zero-shot text embeddings outperform average image embeddings of the labeled source-domain data, so a generic text classifier is more robust than a classifier learned from limited images.

Limitations

• Evaluated only on standard DomainBed image classification benchmarks with CLIP-style teachers.
• Fine-tuning on limited in-domain data degrades OOD performance, so the method relies on fixed text-embedding classifiers.
• Requires text-prompt class names for the classifier head, so it depends on class-name availability.

Experimental setup

Datasets: OfficeHome PACS VLCS Terra-Incognita DomainNet

Benchmarks: OfficeHome PACS VLCS Terra-Incognita DomainNet

Metrics: OOD accuracy in-domain validation accuracy

Baselines: ERM KD SWAD MIRO DART SAGM LP-FT FLYP CLIPood WiSE-FT RISE SimKD

Builds on: CLIP Knowledge distillation (Hinton et al.) SWAD DomainBed RISE

Closest papers by meaning

• CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment (NeurIPS 2024 · similarity 0.90)
• PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization (CVPR 2024 · similarity 0.89)
• Domain Generalization in CLIP via Learning with Diverse Text Prompts (CVPR 2025 · similarity 0.89)
• Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.89)
• Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization (ICLR 2024 · similarity 0.89)
• Bridge the Modality and Capability Gaps in Vision-Language Model Selection (NeurIPS 2024 · similarity 0.89)