Research map / Vision-language and multimodal / CLIP and zero-shot

Joint Vision-Language Social Bias Removal for CLIP

CVPR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Self-distillation for vision-language pretraining · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a joint vision-language debiasing method for CLIP that aligns image and text bias embeddings before removing them, with counterfactual debiasing, and introduces the ABLE metric and out-of-domain fairness evaluation.

Problem

Social bias removal methods for CLIP degrade V-L zero-shot alignment ("over-debiasing") because image and text biases are unbalanced and misaligned when debiased separately or with the same dimensions.

Approach

A frozen-CLIP bias alignment module maps image and text embeddings to bias embeddings whose intra-modal queue-based distributions are aligned via KL divergence; a counterfactual debiasing loss on text pairs with opposite attributes, combined with contrastive learning, removes bias from both modalities while preserving V-L alignment.

Key results

• On ViT-B/16 gender debiasing trained on FairFace, the method reaches ABLE 78.35% vs 73.87% for undebiased CLIP while keeping ImageNet top-1 at 68.05% (vs 68.31%).
• CLIP-clip and Biased-prompts reduce bias but drop ABLE substantially (e.g., Biased-prompts age ViT-B/16 ABLE 54.33% vs 58.94% original), indicating over-debiasing.
• Removing the counterfactual loss or the bias alignment step each degrades debiasing and/or V-L alignment in ablations; joint debiasing of gender, age, and race is also demonstrated.

Measured effects reported in the paper

Surprising findings

• Existing CLIP debiasing methods that achieve large bias reductions can substantially lower zero-shot V-L performance, a trade-off the authors term over-debiasing.
• Image and text social biases are distributed differently across bias types (e.g., gender-career in images vs gender-science and race in text), so debiasing both modalities with the same dimensions is unreliable.

Limitations

• Counterfactual image embeddings are unavailable, so the counterfactual debiasing loss is applied only to text.
• Evaluation focuses on gender, age, and race with face-based datasets; out-of-domain generalization is tested on a limited set of datasets (UTKFace, FACET).

Experimental setup

Datasets: FairFace UTKFace FACET ImageNet-1K Flickr30K (Flickr-1K) SEAT IEAT

Benchmarks: FairFace UTKFace FACET ImageNet-1K Flickr-1K

Metrics: MaxSkew@k NDKL@k Top-1 accuracy Top-5 accuracy Recall@5 ABLE

Baselines: CLIP-clip Biased-prompts Original CLIP

Builds on: CLIP MoCo ALBEF CLIP-clip Biased-prompts

Closest papers by meaning

• Interpretable Debiasing of Vision-Language Models for Social Fairness (CVPR 2026 · similarity 0.90)
• BendVLM: Test-Time Debiasing of Vision-Language Embeddings (NeurIPS 2024 · similarity 0.90)
• SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP (ICLR 2025 · similarity 0.90)
• A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks (NeurIPS 2024 · similarity 0.89)
• Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models (ICLR 2025 · similarity 0.88)
• FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication (CVPR 2024 · similarity 0.88)