Research map / Vision-language and multimodal / CLIP and zero-shot
Representation-Level Counterfactual Calibration for Debiased Zero-Shot Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an inference-only, representation-level counterfactual calibration for CLIP that synthesizes counterfactual image embeddings by recombining object features with alternative contexts and subtracts background-only effects via a Total Direct Effect estimate.
Problem
Object-context shortcuts in vision-language models (CLIP) cause hallucinated zero-shot predictions and worst-group failures when test scenes differ from training co-occurrences.
Approach
Estimate object and background expectations in CLIP's embedding space from token-level ViT direct effects, build counterfactual embeddings by recombining object features with contexts drawn from external datasets, batch neighbors, or text descriptions, and apply a logit-space TDE that subtracts background-only scores, without retraining or prompt design.
Key results
- On Waterbirds with ViT-B/32, the virtual variant reaches 83.69% average accuracy vs. 64.88% for CLIP, and the external variant reaches 79.16% worst-group accuracy vs. 34.55%.
- On COCO-GB v1 (ViT-B/16), the virtual variant reduces the female-male gap to 0.80 from 6.20 for CLIP.
- On UrbanCars (ViT-B/16), the virtual variant reaches 71.87% average accuracy vs. 63.07% for CLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use representation-level counterfactual calibration with virtual contexts instead of vanilla CLIP | CLIP ViT-B/16 · Waterbirds | average accuracy: 77.20 -> 89.47 | Helped |
| use representation-level counterfactual calibration with virtual contexts instead of vanilla CLIP | CLIP ViT-B/16 · COCO-GB v1 | gender gap: 6.20 -> 0.80 | Helped |
| use representation-level counterfactual calibration with virtual contexts instead of vanilla CLIP | CLIP ViT-B/16 · UrbanCars | average accuracy: 63.07 -> 71.87 | Helped |
| add the intervention module (ablation) | Ours (External), CLIP ViT-B/32 · Waterbirds | average accuracy: 75.70 -> 79.96 | Helped |
| add the TDE module (ablation) | Ours (External), CLIP ViT-B/32 · Waterbirds | average accuracy: 76.10 -> 79.96 | Helped |
| add the sampler (ablation) | Ours (External), CLIP ViT-B/32 · Waterbirds | average accuracy: 79.92 -> 79.96 | No clear effect |
| replace semantic contexts with sampler-filtered random vectors (ablation) | Ours · Waterbirds | average accuracy: 65.19; 59.94; 77.36; 69.55 | Mixed |
Surprising findings
- Random context vectors ablated through the sampler sometimes underperform CLIP (e.g., ViT-B/16 Waterbirds 59.94 vs 77.20), indicating semantically specific contexts are needed.
- TBD, a fine-tuning-based debiasing method, substantially degrades in-distribution accuracy on UrbanCars (58.00 I.D.).
Limitations
- Results are reported from a fixed random seed with best-round selection, without error bars.
- Sensitivity to context pool choice and hyperparameters (lambda, lambda-hat, alpha) is discussed only in the appendix.
Experimental setup
Datasets: Waterbirds UrbanCars COCO-GB v1 NICO Places-365
Benchmarks: Waterbirds COCO-GB v1 UrbanCars NICO
Metrics: average accuracy worst-group accuracy gender accuracy gap
Baselines: CLIP TBD PC PC+ B2T ALIA Mixup CutMix AugMix Cutout MEMO
Builds on: CLIP Total Direct Effect (counterfactual VQA) Projecting out Prompt Confounding (PC) TBD B2T
Closest papers by meaning
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.87)
- Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization (ICML 2025 · similarity 0.87)
- Conformal Prediction for Zero-Shot Models (CVPR 2025 · similarity 0.87)
- T2ICount: Enhancing Cross-modal Understanding for Zero-Shot Counting (CVPR 2025 · similarity 0.86)
- PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts (ICLR 2024 · similarity 0.86)
- On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do We Really Need Prompt Learning? (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.