Research map / Vision-language and multimodal / CLIP and zero-shot
Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoOPood, a decoupled prompt tuning method for CLIP that splits image features into invariant and spurious parts via two projection layers and aligns text only with the invariant part, improving worst-group OOD accuracy on spurious-correlation benchmarks.
Problem
Prompt tuning of vision-language models like CLIP aligns text uniformly with image features, so spurious correlations from biased training data get absorbed, hurting OOD generalization under distribution shift.
Approach
Decompose frozen CLIP image features into invariant and spurious components with two lightweight projection layers, then run two independent contrastive phases: pull text embeddings toward the invariant image embedding (cross-entropy) and push them toward uniform prediction on the spurious embedding (KL), with a conditional mutual information penalty encouraging independence.
Key results
- On Waterbirds with ResNet-50, CoOPood reaches 82.38% average and 60.28% worst-group accuracy, versus 78.98% / 48.91% for CoOp, without group labels.
- On ImageNet-1K baby pacifier subset with ViT-B/32, CoOPood improves worst-group accuracy to 90.72% from 88.66% for ERM and CoOp.
- Plugging CoOPood into CoCoOp raises Waterbirds worst-group accuracy from 52.26% to 60.97% (ResNet-50).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace CoOp's uniform alignment with decoupled alignment | CLIP prompt tuning · Waterbirds, worst group (ResNet-50) | accuracy: 48.91 -> 60.28 | Helped |
| replace CoOp's uniform alignment with decoupled alignment | CLIP prompt tuning · CelebA, worst group (ResNet-50) | accuracy: 26.11 -> 31.11 | Helped |
| replace CoOp's uniform alignment with decoupled alignment | CLIP prompt tuning · ImageNet-1K, worst group (ResNet-50) | accuracy: 78.87 -> 86.08 | Helped |
| replace CoOp's uniform alignment with decoupled alignment | CLIP prompt tuning · Waterbirds, worst group (ViT-B/32) | accuracy: 43.93 -> 53.74 | Helped |
| replace CoOp's uniform alignment with decoupled alignment | CLIP prompt tuning · CelebA, worst group (ViT-B/32) | accuracy: 25.56 -> 27.22 | Helped |
| replace CoOp's uniform alignment with decoupled alignment | CLIP prompt tuning · ImageNet-1K, worst group (ViT-B/32) | accuracy: 88.66 -> 90.72 | Helped |
| use CoOPood rather than CoCoOp | CLIP prompt tuning · Waterbirds (ResNet-50) | worst-group accuracy: 52.26 -> 60.97 | Helped |
| use CoOPood rather than CoCoOp | CLIP prompt tuning · CelebA (ResNet-50) | worst-group accuracy: 61.11 -> 70.64 | Helped |
| add the conditional mutual information loss Lc (ablation) | CoOPood · Waterbirds (ResNet-50) | average accuracy: 81.43 -> 82.38 | Helped |
| add the conditional mutual information loss Lc (ablation) | CoOPood · Waterbirds (ResNet-50) | worst-group accuracy: 57.65 -> 60.28 | Helped |
| add the spurious-feature loss Ls (ablation) | CoOPood · Waterbirds (ResNet-50) | average accuracy: 81.72 -> 82.38 | Helped |
| add the spurious-feature loss Ls (ablation) | CoOPood · Waterbirds (ResNet-50) | worst-group accuracy: 58.53 -> 60.28 | Helped |
Surprising findings
- On CelebA, the pre-trained CLIP achieves higher worst-group accuracy than CoOPood and CoOp, which the authors attribute partly to dataset properties and training split.
- Group-label-free CoOPood underperforms Group DRO on Waterbirds and CelebA worst-group accuracy, so the gains depend on group information for the strongest results.
Limitations
- CelebA worst-group accuracy does not improve over pre-trained CLIP
- Best results require group labels (Group CoOPood), which are not available in the standard setting
- Hyperparameters alpha and beta are set per dataset; evaluation limited to image classification
Experimental setup
Datasets: Waterbirds CelebA ImageNet-1K PACS ImageNet-A ImageNet-R Food101 Flower102
Benchmarks: Waterbirds CelebA ImageNet-1K PACS ImageNet-A ImageNet-R Food101 Flower102
Metrics: average accuracy worst-group accuracy
Baselines: Pre-trained CLIP CoOp ERM Group DRO CoCoOp DFR PDE SDM TextSpan RoboShot
Builds on: CoOp CLIP Group DRO CoCoOp
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.92)
- O-TPT: Orthogonality Constraints for Calibrating Test-time Prompt Tuning in Vision-Language Models (CVPR 2025 · similarity 0.91)
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.90)
- Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models (ICLR 2024 · similarity 0.90)
- LAMM: Label Alignment for Multi-Modal Prompt Learning (AAAI 2024 · similarity 0.90)
- Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.