Research map / Vision-language and multimodal / CLIP and zero-shot
Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Contrastive Feature Recalibration (CFR), which fine-tunes only CLIP's projection head with a contrastive loss on a calibration set of pre-trained-CLIP-misclassified samples, to improve worst-group accuracy without group annotations.
Problem
Spurious correlations cause CLIP-based classifiers to have low worst-group accuracy on minority groups, but existing mitigation methods need group annotations or dual-phase training; the paper asks how to improve group robustness of CLIP efficiently without group labels.
Approach
Build a calibration set from training samples misclassified by a pretrained (projection-head-tuned) CLIP, then recalibrate frozen-backbone features of these anchors with a contrastive loss that pulls samples toward an EMA-estimated class centroid and pushes them from negative-class samples, combined with a cosine-similarity loss over the full training set.
Key results
- CFR with DPS+RNS on CLIP-ResNet improves WGA over AFR, JTT, and CnC by 15.25–28.55, 9.77–20.22, and 8.55–9.34 points respectively across datasets.
- With CLIP-ViT, CFR (DPS+RNS) achieves up to 3.74–14.81 WGA points over the three semi-supervised baselines.
- Adding the cosine-similarity loss over the full training set improves WGA by 7.26 points for ResNet-50 with DPS+RNS (76.93 vs 69.67).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CFR with DPS+RNS instead of the semi-supervised baselines | CLIP ResNet-50 · Waterbirds, CelebA, CheXpert, and MetaShift | WGA | Mixed |
| Add dynamic positive centroid sampling (DPS) instead of RPS (ablation) | CFR with RNS on CLIP ResNet-50 · Waterbirds | WGA: 73.08 -> 76.93 | Helped |
| Add dynamic positive centroid sampling (DPS) instead of RPS (ablation) | CFR with RNS on CLIP ResNet-50 · CelebA | WGA: 72.78 -> 73.66 | Helped |
| Add dynamic positive centroid sampling (DPS) instead of RPS (ablation) | CFR with RNS on CLIP ResNet-50 · CheXpert | WGA: 49.50 -> 54.44 | Helped |
| Add dynamic positive centroid sampling (DPS) instead of RPS (ablation) | CFR with RNS on CLIP ResNet-50 · MetaShift | WGA: 80.13 -> 81.54 | Helped |
| Use RNS instead of NNS for negative sample selection (ablation) | CFR with DPS on CLIP ResNet-50 · Waterbirds | WGA: 76.63 -> 76.93 | Helped |
| Use RNS instead of NNS for negative sample selection (ablation) | CFR with DPS on CLIP ResNet-50 · CelebA | WGA: 73.21 -> 73.66 | Helped |
| Use RNS instead of NNS for negative sample selection (ablation) | CFR with DPS on CLIP ResNet-50 · CheXpert | WGA: 52.67 -> 54.44 | Helped |
| Use RNS instead of NNS for negative sample selection (ablation) | CFR with DPS on CLIP ResNet-50 · MetaShift | WGA: 81.54 -> 81.54 | No clear effect |
| Remove holistic data integration loss LCS (ablation) | CFR DPS+RNS · Waterbirds | WGA: 76.93 -> 69.67 | Hurt |
| Remove holistic data integration loss LCS (ablation) | CFR DPS+RNS · Waterbirds | WGA: 88.23 -> 87.07 | Hurt |
Surprising findings
- The paper reports that NNS negative sampling, usually expected to help, underperforms random negative sampling on CLIP-ResNet, which it attributes to sensitivity to k and batch size.
Limitations
- Performance gap remains relative to supervised methods on ResNet backbone.
- NNS sampling is computationally heavy and sensitive to k and batch size.
- Calibration-set selection relies on misclassification by a pretrained model, with alternatives left for future work.
Experimental setup
Datasets: Waterbirds CelebA CheXpert MetaShift
Benchmarks: Waterbirds CelebA CheXpert MetaShift
Metrics: worst-group accuracy (WGA) average accuracy
Baselines: ERM GroupDRO S-CS S-CL DFR AFR JTT CnC Con-Adapter
Builds on: CLIP Deep Feature Reweighting (DFR) Just Train Twice (JTT) Correct-n-Contrast (CnC) Simple contrastive learning (SimCLR-style)
Closest papers by meaning
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
- Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling (ICLR 2025 · similarity 0.90)
- Demystifying CLIP Data (ICLR 2024 · similarity 0.89)
- Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models (ICML 2024 · similarity 0.89)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (CVPR 2024 · similarity 0.89)
- UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.