Research map / Label-efficient and robust learning / OOD and anomaly detection
Rethinking the Evaluation Protocol of Domain Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-examines the DomainBed domain generalization protocol, arguing that ImageNet supervised pretraining and oracle model selection leak test-domain information, and proposes self-supervised (or scratch) initialization and multiple test domains with new leaderboards for ten algorithms.
Problem
Current domain generalization evaluation protocol (DomainBed) may leak test data information through ImageNet supervised pretraining and oracle model selection on a single test domain, so it may not accurately measure OOD generalization.
Approach
Shows that supervised ImageNet pretraining makes linear probing match or beat fine-tuning on real-photo test domains, and that oracle model selection leakage shrinks when multiple test domains are used; proposes self-supervised (MoCo-v2/v3) or scratch initialization and multi-domain testing, with re-run leaderboards.
Key results
- Supervised ImageNet-pretrained ResNet-50 linear probing matches or beats fine-tuning on several real-photo test domains (e.g., PACS-Photo LP 97.7 vs FT 97.4 avg LP 67.3 vs FT 85.3 overall, OfficeHome avg LP 67.9 vs FT 65.5).
- Rank correlation with training from scratch rises from 0.261 (supervised pretraining) to 0.576 (MoCo-v2) and 0.600 (MoCo-v3).
- Using K=3 test domains reduces oracle-selection leakage by 35% on DomainNet and 79% on NICO++ (autumn, rock, dim) relative to K=1 in the reported comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the number of test domains used for oracle model selection from K=1 to K=3 (ablation) | Oracle model selection on DomainNet and NICO++ · DomainNet and NICO++ | oracle-selection leakage: -35%; -38%; -42%; -79% | Helped |
| replace supervised pretraining with self-supervised pretraining (ablation) | ResNet-50 and ViT-B/16 on OfficeHome · OfficeHome | LP versus FT test accuracy pattern | Helped |
| change the backbone architecture from ResNet-50 to ResNet-18 or ViT-B/16 while keeping supervised pretraining (ablation) | OfficeHome linear probing and fine-tuning · OfficeHome | LP versus FT test accuracy pattern | No clear effect |
| use supervised-pretrained ResNet-50 linear probing rather than fine-tuning (ablation) | Domain generalization evaluation · PACS average | accuracy: 85.3±0.6 -> 67.3±0.3 | Hurt |
| use supervised-pretrained ResNet-50 linear probing rather than fine-tuning (ablation) | Domain generalization evaluation · VLCS average | accuracy: 77.1±1.0 -> 76.0±0.6 | Hurt |
| use supervised-pretrained ResNet-50 linear probing rather than fine-tuning (ablation) | Domain generalization evaluation · OfficeHome average | accuracy: 65.5±0.2 -> 67.9±0.2 | Helped |
| use supervised-pretrained ResNet-50 linear probing rather than fine-tuning (ablation) | Domain generalization evaluation · Terra Incognita average | accuracy: 48.1±3.1 -> 37.6±1.7 | Hurt |
| use self-supervised-pretrained ResNet-50 rather than supervised-pretrained ResNet-50 (ablation) | ERM domain generalization evaluation · OfficeHome | average accuracy: 66.5±0.3 -> 57.3±0.3 | Hurt |
| use MoCo-v2 pretrained ResNet-50 rather than supervised-pretrained ResNet-50 (ablation) | SWAD domain generalization evaluation · DomainNet | accuracy: 46.5±0.1 -> 42.9±0.1 | Hurt |
| use MoCo-v3 pretrained ViT-B/16 rather than supervised-pretrained ResNet-50 (ablation) | SWAD domain generalization evaluation · DomainNet | accuracy: 46.5±0.1 -> 49.6±0.1 | Helped |
| use SWAD rather than ERM | Domain generalization evaluation · PACS, supervised-pretrained ResNet-50 | accuracy: 85.5±0.2 -> 88.1±0.1 | Helped |
| use SWAD rather than ERM | Domain generalization evaluation · PACS, MoCo-v2 pretrained ResNet-50 | accuracy: 84.1±0.3 -> 86.2±0.6 | Helped |
Surprising findings
- Linear probing on supervised ImageNet features often outperforms full fine-tuning on real-photo test domains, which the authors interpret as test-data leakage from pretraining.
- RSC ranks 2nd on the self-supervised leaderboards despite failing to beat ERM under supervised pretraining.
Limitations
- Self-supervised pretraining still may carry some leakage; training from scratch is only feasible on DomainNet and NICO++.
- Leaderboards exclude TerraIncognita due to large random fluctuations.
- Multi-domain oracle analysis is limited to datasets with enough domains (DomainNet, NICO++).
Experimental setup
Datasets: PACS VLCS OfficeHome TerraIncognita DomainNet NICO++
Benchmarks: PACS VLCS OfficeHome DomainNet NICO++
Metrics: test accuracy Spearman rank correlation ranking change
Baselines: ERM SWAD RSC GroupDRO Fishr CORAL MMD SagNet IRM Mixup
Builds on: DomainBed MoCo-v2 MoCo-v3 SWAD
Closest papers by meaning
- In Search of Forgotten Domain Generalization (ICLR 2025 · similarity 0.90)
- Lost Domain Generalization Is a Natural Consequence of Lack of Training Domains (AAAI 2024 · similarity 0.88)
- The Value of Out-of-Distribution Data (ICML 2023 · similarity 0.88)
- OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable? (ICML 2025 · similarity 0.86)
- Domain Generalisation via Imprecise Learning (ICML 2024 · similarity 0.86)
- CIFAR-10-Warehouse: Broad and More Realistic Testbeds in Model Generalization Analysis (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.