Research map / Label-efficient and robust learning / OOD and anomaly detection

In Search of Forgotten Domain Generalization

ICLR 2025 · Label-efficient and robust learning › OOD and anomaly detection · approach: Out-of-distribution generalization theory · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds clean natural and rendition subsets of LAION via domain classifiers and shows CLIP's rendition OOD performance is largely driven by in-domain rendition training data, with optimal natural/rendition mixing ratios around 1:3 to 1:1.

Problem

Web-scale training sets (e.g., LAION) contain many images from test domains (renditions), so CLIP's strong OOD performance on ImageNet-Sketch, ImageNet-R, and DomainNet may reflect domain contamination rather than genuine OOD generalization.

Approach

Train a domain classifier (fine-tuned CLIP ViT-L/14 ternary classifier and density-ratio binary classifier) on manually labeled LAION data, use it to subsample domain-pure LAION-Natural (~57M) and LAION-Rendition (~16M) datasets, and also clean OOD test sets, then train CLIP from scratch on controlled domain mixtures to measure true OOD generalization.

Key results

• CLIP trained only on LAION-Natural (~57M) reaches 17.81% average rendition accuracy versus 39.58% for LAION-Rand-57M, with natural accuracy roughly unchanged.
• Relative corrected OOD accuracy is around or above 1.0 on natural test sets but drops to around 0.4 for most rendition test sets.
• Mixing 13M or 16M renditions into LAION-Natural nearly doubles rendition-domain accuracy (about 40-42%) with minimal natural-domain impact; a rendition-to-natural ratio between 1:3 and 1:1 gives the best overall performance.

Measured effects reported in the paper

Surprising findings

• Widely used OOD benchmarks such as ImageNet-R and DomainNet-Real/Clipart/Painting/Infograph are at least 5% contaminated with images from the opposite domain.
• The 'combined-pruned' LAION subset from prior work still contains at least 6.4% renditions (about 11 million images), so its reported OOD gains may be inflated.

Limitations

• Restricted to natural vs. rendition domains; other domain shifts not studied
• Training scale limited to 4M-57M samples due to availability of rendition images (16M in LAION-Rendition)
• Domain labels are from a single annotator with a texture-centric labeling handbook, and classifiers have imperfect recall

Experimental setup

Datasets: LAION-200M LAION-400M ImageNet-Train ImageNet-Val ImageNet-V2 ImageNet-A ImageNet-R ImageNet-Sketch ObjectNet DomainNet

Benchmarks: ImageNet-Val ImageNet-V2 ImageNet-A ObjectNet ImageNet-R ImageNet-Sketch DomainNet-Real DomainNet-Painting DomainNet-Clipart DomainNet-Infograph DomainNet-Sketch DomainNet-Quickdraw

Metrics: top-1 accuracy precision recall relative corrected OOD accuracy effective robustness

Baselines: CSD DR CE LAION-Rand-57M combined-pruned (Mayilvahanan et al. 2024)

Builds on: CLIP LAION-400M density ratio domain classification Contrastive Style Descriptors DomainNet

Closest papers by meaning

• Rethinking the Evaluation Protocol of Domain Generalization (CVPR 2024 · similarity 0.90)
• LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models (ICML 2025 · similarity 0.89)
• CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.88)
• Towards Robust Out-of-Distribution Generalization Bounds via Sharpness (ICLR 2024 · similarity 0.88)
• Unknown Prompt the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization (CVPR 2024 · similarity 0.87)
• A Bayesian Approach to OOD Robustness in Image Classification (CVPR 2024 · similarity 0.87)