Research map / Label-efficient and robust learning / Domain adaptation and generalization
Is Large-scale Pretraining the Secret to Good Domain Generalization?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Alignment Hypothesis, which links pretraining image-text alignment to multi-source DG accuracy, and uses it to split DomainBed into IP and OOP subsets to show current CLIP-based DG methods fail on low-alignment data.
Problem
Unclear whether recent gains in multi-source domain generalization come from better DG finetuning or merely from stronger web-scale CLIP pretraining, and how well DG methods handle target data that pretraining did not align well.
Approach
Final DG accuracy is predicted by the pre-finetuning cosine alignment between image and class-label text embeddings (AlignmentScore), which is a stronger predictor than perceptual nearest-neighbor similarity to pretraining data; this score is used to split DomainBed test data into In-Pretraining (IP) and Out-of-Pretraining (OOP) subsets.
Key results
- CLIPood reaches 84.7% on DomainBed-IP but only 57.1% on DomainBed-OOP, versus 78.1% on unsplit DomainBed-All.
- MIRO + MPA achieves the best OOP average of 57.2%, slightly above CLIPood, while ERM averages 47.0% on OOP.
- On OfficeHome and PACS IP splits, the best DG method exceeds the oracle trained on target data; on TerraIncognita CLIPood is about 30% below the oracle.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate on high-alignment IP rather than low-alignment OOP samples | CLIPood · DomainBed | accuracy: 57.1% -> 84.7% | Helped |
| Use MPA instead of ERM | Domain Generalization · DomainBed-OOP Average | accuracy: 47.0 -> 53.0 | Helped |
| Combine MIRO with MPA | Domain Generalization · DomainBed-OOP Average | accuracy: 53.0 -> 57.2 | Helped |
| Use MIRO + MPA rather than CLIPood | Domain Generalization · DomainBed-OOP Average | accuracy: 57.1 -> 57.2 | No clear effect |
| Use MIRO rather than MPA | Domain Generalization · DomainBed-OOP Average | accuracy: 53.0 -> 52.4 | Hurt |
| Use SWAD rather than MPA | Domain Generalization · DomainBed-OOP Average | accuracy: 53.0 -> 51.1 | Hurt |
| Finetune rather than use zero-shot predictions | CLIP-based DG methods · PACS-IP and VLCS-IP | accuracy | Hurt |
| Use pretrained CLIP rather than train ResNet-50 from scratch | MPA · DomainBed-All Average | accuracy: 28.0 -> 73.6 | Helped |
| Use alignment score instead of perceptual similarity to stratify evaluation samples | DG evaluation · DomainBed | accuracy | Helped |
| Increase pretraining alignment of the evaluated samples | DG methods · DomainBed-IP versus DomainBed-OOP | accuracy | Helped |
| Use calibrated zero-shot confidence instead of AlignmentScore | Generalization prediction · OfficeHome and DomainNet | predictive alignment across datasets | Hurt |
Surprising findings
- On the IP subset, zero-shot OpenCLIP exceeds fine-tuned DG models on PACS and VLCS, suggesting catastrophic forgetting of pretrained knowledge.
- Training ResNet-50 from scratch yields only 28.0% average accuracy versus 73.6% for CLIP-based MPA, showing DG methods depend heavily on pretraining.
Limitations
- The IP/OOP threshold of 0.21 is a subjective choice informed by observed trends.
- Analysis is restricted to CLIP-style image-text backbones and five DomainBed datasets; other backbones tested only on two datasets with two methods.
- Samples with AlignmentScore below 0.15 or above 0.4 are discarded, which removes a portion of data due to label noise and OCR.
Experimental setup
Datasets: VLCS PACS OfficeHome TerraIncognita DomainNet LAION-400M
Benchmarks: DomainBed-IP DomainBed-OOP DomainBed-All (VLCS, PACS, OfficeHome, TerraIncognita, DomainNet)
Metrics: top-1 accuracy
Baselines: ERM CORAL SAGM LP-FT SWAD MIRO VL2V-SD CLIPood MPA Attention Tuning
Builds on: CLIP DomainBed Image Similarity Hypothesis MIRO CLIPood
Closest papers by meaning
- PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization (CVPR 2024 · similarity 0.89)
- Rethinking Multi-domain Generalization with A General Learning Objective (CVPR 2024 · similarity 0.89)
- Towards Generalizing to Unseen Domains with Few Labels (CVPR 2024 · similarity 0.88)
- Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gap (ICLR 2025 · similarity 0.88)
- Improving Domain Generalization with Domain Relations (ICLR 2024 · similarity 0.88)
- Domain Generalization in CLIP via Learning with Diverse Text Prompts (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.