Research map / Vision-language and multimodal / CLIP and zero-shot
Rethinking Misalignment in Vision-Language Model Adaptation from a Causal Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CDC, a causal-inference-based method that uses front-door adjustment with multiple decoupled prompt templates and Dempster-Shafer evidential fusion to mitigate data misalignment when adapting CLIP via prompt tuning.
Problem
CLIP adapted to downstream tasks via soft prompt tuning overfits base classes under data misalignment (label and distribution inconsistency between training and testing), hurting generalization to new classes.
Approach
Builds a structural causal model showing task-irrelevant generative factors act as unobservable confounders, and applies front-door adjustment using task-relevant semantics as mediators. Semantics are decoupled via multiple prompt templates with augmentation-specific image inputs and orthogonality losses, and predictions from each template are fused with Dempster-Shafer evidence theory weighted by
Key results
- Base-to-new setting: CDC reaches 80.25 HM averaged over 11 datasets, +1.70 over MaPLe (78.55), with new-class accuracy 77.38 vs 75.14.
- Cross-dataset: CDC averages 66.73 across 10 target datasets vs 66.30 for MaPLe, a +0.43 gain.
- Cross-domain: CDC averages 60.93 vs 60.28 for MaPLe across ImageNet variants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt CLIP with CDC instead of MaPLe | CLIP ViT-B/16 · Base-to-new Avg | HM accuracy: 78.55 -> 80.25; +1.70 | Helped |
| Adapt CLIP with CDC instead of MaPLe | CLIP ViT-B/16 · Cross-dataset target Avg | accuracy: 66.30 -> 66.73; 0.43% | Helped |
| Adapt CLIP with CDC instead of MaPLe | CLIP ViT-B/16 · Cross-domain target Avg | accuracy: 60.28 -> 60.93; 0.65% | Helped |
| Add multiple templates (M=4) with averaged predictions (ablation) | MaPLe · Base-to-new Avg | HM accuracy: 78.55 -> 79.36; 0.81% | Helped |
| Add DSTC to four templates (ablation) | Four-template model · Base-to-new Avg | HM accuracy: 79.36 -> 79.61; 0.25% | Helped |
| Add image-branch VSD without text-branch VSD (ablation) | Four templates + DSTC · Base-to-new Avg | HM accuracy: 79.61 -> 79.90 | Helped |
| Add text-branch VSD without image-branch VSD (ablation) | Four templates + DSTC · Base-to-new Avg | HM accuracy: 79.61 -> 79.77 | Helped |
| Add VSD in both image and text branches (ablation) | Four templates + DSTC · Base-to-new Avg | HM accuracy: 79.61 -> 80.25; 0.64% | Helped |
| Increase template count from M=1 to M=2 (ablation) | Model trained with trusted cross-entropy only · Base-to-new Avg | HM accuracy: 77.00 -> 79.57 | Helped |
| Increase template count from M=4 to M=8 (ablation) | Model trained with trusted cross-entropy only · Base-to-new Avg | HM accuracy: 79.61 -> 80.23 | Helped |
| Increase the weight of the decoupling loss to β=5 or β=10 rather than β=0 (ablation) | CDC · DTD and Aircraft | HM accuracy | Helped |
| Increase the decoupling-loss weight to β=20 (ablation) | CDC · DTD and Aircraft | HM accuracy | Hurt |
Surprising findings
- Gains from multiple templates persist even without explicit decoupling, which the authors attribute to different initializations learning distinct text embeddings.
- Performance on ImageNet-A slightly decreases relative to MaPLe (50.40 vs 50.90).
Limitations
- Increased inference and training cost because each image is encoded once per template.
- Hyperparameters (beta, gamma, M, augmentation choice) need tuning and augmentation choice has notable impact, especially on fine-grained datasets like Cars and Aircraft.
- Improvements over MaPLe in cross-dataset and cross-domain settings are modest (about 0.4-0.7 points average).
Experimental setup
Datasets: ImageNet Caltech101 Oxford Pets Stanford Cars Flowers102 Food101 FGVC Aircraft SUN397 DTD EuroSAT UCF-101
Benchmarks: ImageNet Caltech101 Oxford Pets Stanford Cars Flowers102 Food101 FGVC Aircraft SUN397 DTD EuroSAT UCF-101 ImageNetV2
Metrics: accuracy harmonic mean
Baselines: CoOp CoCoOp MaPLe CLIP
Builds on: CLIP MaPLe CoOp CoCoOp Dempster-Shafer evidence theory
Closest papers by meaning
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.89)
- Vision-Language Models Create Cross-Modal Task Representations (ICML 2025 · similarity 0.88)
- Unveiling the Tapestry of Consistency in Large Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models (ICML 2024 · similarity 0.88)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.88)
- Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.