Research map / Label-efficient and robust learning / Domain adaptation and generalization

When and How Does CLIP Enable Domain and Compositional Generalization?

ICML 2025 · Label-efficient and robust learning › Domain adaptation and generalization · approach: Augmentation and mixture-of-experts generalization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains CLIP models on controlled mixtures of DomainNet domains and ImageNet-Captions to test when CLIP generalizes to unseen domains and unseen classes within partially seen domains, and analyzes why with SAE feature sharing, CKA, and a new mechanistic circuit similarity measure.

Problem

Whether CLIP generalizes to an entirely unseen domain when trained on a diverse mixture of domains (domain generalization) and to unseen classes within partially seen domains (compositional generalization), and which training factors and internal mechanisms govern this.

Approach

Systematically construct training distributions with controlled domain diversity and class exposure (natural-only, leave-out-domain, CG low/high-diversity) so that only the domain mixture varies, then analyze the resulting models with data-centric analyses, sparse autoencoder feature sharing, representational similarity (CKA), and a new mechanistic (circuit) similarity measure.

Key results

• CG high-diversity on Sketch reached 28.1% balanced top-1 vs 30.1% for Leave-out-domain, with an upper bound of 44.3% when trained on classes C2 of the test domain.
• Adding ImageNet-Sketch sketches of non-queried classes raised Sketch compositional generalization from 28.1% to 36.9% in the CG high-diversity setting.
• Domain diversity raised SAE top-k shared feature overlap by +7.1 points on Clipart and +4.1 on Sketch (Natural-only to Leave-out-domain).

Measured effects reported in the paper

Surprising findings

• Seeing a subset of the test domain's classes can make compositional generalization worse than not seeing the test domain at all (e.g., Sketch 19.2 CG low-diversity vs 30.1 Leave-out-domain).
• Quickdraw fails to generalize to unseen classes even with high domain diversity, and removing domain-specific captions does not fix it.

Limitations

• Experiments limited to small-scale datasets (0.5M to 10M samples) and ResNet-50 CLIP, without large-scale domain data
• Domain generalization gap to upper bound persists; CLIP only weakly generalizes
• Dataset quality effects not studied

Experimental setup

Datasets: ImageNet-Captions CC3M CC12M DomainNet ImageNet-Sketch

Benchmarks: DomainNet

Metrics: balanced top-1 accuracy balanced top-5 accuracy macro F1 effective robustness CKA Jaccard index of circuit components normalized Weisfeiler-Lehman subtree kernel

Baselines: Natural-only CLIP Leave-out-domain CLIP CG low-diversity CLIP CG high-diversity CLIP supervised ResNet-50 classifier SigLIP BLIP

Builds on: CLIP OpenCLIP DomainNet ImageNet-Captions Sparse autoencoders

Closest papers by meaning

• Domain Generalization in CLIP via Learning with Diverse Text Prompts (CVPR 2025 · similarity 0.91)
• CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment (NeurIPS 2024 · similarity 0.91)
• Does Data Scaling Lead to Visual Compositional Generalization? (ICML 2025 · similarity 0.90)
• Unknown Prompt the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization (CVPR 2024 · similarity 0.89)
• Does CLIP’s generalization performance mainly stem from high train-test similarity? (ICLR 2024 · similarity 0.89)
• When does compositional structure yield compositional generalization? A kernel theory. (ICLR 2025 · similarity 0.89)