Research map / Vision-language and multimodal / CLIP and zero-shot
Learning Invariant Causal Mechanism from Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLIP-ICM, which learns a linear projection of CLIP embeddings onto an invariant subspace using interventional data (image augmentations or text edits) and predicts within that subspace to improve CLIP's OOD generalization without backbone retraining.
Problem
Fine-tuning CLIP for downstream tasks degrades its OOD generalization under domain shift and its zero-shot ability on unseen classes.
Approach
A structural causal model shows that predictions relying only on invariant factors have stable causal mechanisms across environments; the paper proves CLIP embeddings are a linear mix of invariant and variant factors, so a linear map to the invariant subspace can be estimated from interventional data (augmented images or modified captions) with an objective enforcing invariance and reconstruction.
Key results
- CLIP-ICM achieves 76.4% average accuracy on DomainBed with domain shift, and CLIP-ICM Linear-Probe reaches 80.6% versus CLIPood's 78.6%.
- On TerraIncognita, CLIP-ICM Linear-Probe reaches 66.5% versus CLIPood's 60.4%.
- On new classes across OfficeHome and DomainNet, CLIP-ICM averages above the best baseline (CoOp), per the paper's claim of a 5.5% gain.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLIP-ICM Linear Probe instead of CLIPood | CLIP · Terra Incognita | accuracy: surpasses the best-performing method, CLIPOOD, by 6% | Helped |
| Use CLIP-ICM on new classes instead of CoOp | CLIP · OfficeHome and DomainNet New classes | accuracy: 5.5% | Helped |
| Use CLIP-ICM on base classes | CLIP · OfficeHome and DomainNet Base classes | accuracy | Mixed |
| Use image-based rather than text-based interventional data (ablation) | CLIP-ICM · DomainBed | accuracy | Helped |
| Combine image- and text-based interventional data rather than using either alone (ablation) | CLIP-ICM · DomainBed | accuracy | Helped |
| Train a linear probe on projected image embeddings (ablation) | CLIP-ICM · DomainBed average | accuracy: 76.4$_{\pm 0.3}$ -> 80.6$_{\pm 0.2}$ | Helped |
| Train a linear probe on projected image embeddings (ablation) | CLIP-ICM · Terra Incognita | accuracy: 52.5$_{\pm 0.4}$ -> 66.5$_{\pm 0.1}$ | Helped |
Surprising findings
- CLIP-ICM with image-only interventional data (CLIP-ICM*) underperforms on TerraIncognita (49.9%) compared with zero-shot CLIP (34.2%) only in some settings, and its non-linear-probe variant lags the linear-probe variant on most datasets.
- Image-based interventional data outperforms text-based interventional data, while combining both gives the best results.
Limitations
- Theoretical guarantees rely on idealized assumptions (injective generator, condition on D+1 text pairs) that may not hold in practice.
- Interventional data generation via augmentations or LLM-modified captions is an approximation of true interventions.
- Plain CLIP-ICM without linear probe underperforms some baselines on TerraIncognita and DomainNet subsets.
Experimental setup
Datasets: PACS VLCS OfficeHome TerraIncognita DomainNet
Benchmarks: PACS VLCS OfficeHome TerraIncognita DomainNet
Metrics: accuracy
Baselines: Zero-shot CLIP Linear-Probe MIRO CoOp CoCoOp CLIP-Adapter DPL CLIPood
Builds on: CLIP CLIPood CoOp MIRO
Closest papers by meaning
- A Causal Marriage between VLM and IRM from Understanding to Reasoning (CVPR 2026 · similarity 0.91)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.88)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.87)
- Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations (CVPR 2024 · similarity 0.87)
- CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.