Research map / Vision-language and multimodal / Human and embodied modeling

SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples

CVPR 2024 · Vision-language and multimodal › Human and embodied modeling · approach: Scene graph generation · 14 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses Stable Diffusion with cross-attention control to generate 171k counterfactual image-text pairs for probing and mitigating intersectional (race, gender, physical characteristics) social biases in CLIP-style VLMs.

Problem

Existing bias probes for vision-language models use real image-text pairs and examine one social attribute at a time, missing intersectional biases because exhaustive real data for attribute combinations is hard to collect.

Approach

Generate sets of near-identical counterfactual images differing only in intersectional social attributes using Prompt-to-Prompt cross-attention control in Stable Diffusion, followed by three-stage filtering (CLIP image-text similarity, NSFW, CLIP attribute detectability), and use the resulting MaxSkew metric and fine-tuning to probe and reduce skew.

Key results

• All six evaluated VLMs show significant intersectional skew in retrieval; CLIP and FLAVA have lowest overall MaxSkew@K.
• Fine-tuning CLIP on SocialCounterfactuals reduces (Physical Char., Race) MaxSkew@K from 1.19 to 0.87 on held-out occupations, and ALIP drops up to 0.426 absolute.
• Debiased CLIP lowers MaxSkew@K by 15% on VisoGender and 12% on PATA real-image datasets, with minor drops in zero-shot ImageNet accuracy (59.25 to 55.38).

Measured effects reported in the paper

Surprising findings

• Debiasing on synthetic counterfactuals transfers to real-image bias benchmarks (VisoGender, PATA) despite no real training data.
• Gender bias direction differs across VLMs for the same race: OpenCLIP favors Asian male doctors while others favor Indian male doctors.

Limitations

• Binary gender and six race categories only; perceived attributes rather than identity.
• Debiasing causes small drops in zero-shot and retrieval performance.
• Templates and generator may contain latent biases.

Experimental setup

Datasets: SocialCounterfactuals (generated) PATA VisoGender Flickr30K CIFAR10 CIFAR100 Caltech256 ImageNet

Benchmarks: SocialCounterfactuals VisoGender PATA Flickr30K CIFAR10 CIFAR100 Caltech256 ImageNet

Metrics: MaxSkew@K Skew@K recall@K zero-shot accuracy FID IS

Baselines: CLIP ALIP FLAVA LaCLIP OpenCLIP SLIP VisoGender PATA

Builds on: Prompt-to-Prompt Stable Diffusion Skew@K / MaxSkew@K (Geyik et al.) Brooks et al. image pair generation

Closest papers by meaning

• Finetuning Text-to-Image Diffusion Models for Fairness (ICLR 2024 · similarity 0.88)
• OASIS Uncovers: High-Quality T2I Models, Same Old Stereotypes (ICLR 2025 · similarity 0.87)
• Interpretable Debiasing of Vision-Language Models for Social Fairness (CVPR 2026 · similarity 0.87)
• SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models (AAAI 2024 · similarity 0.87)
• Leveraging Diffusion Perturbations for Measuring Fairness in Computer Vision (AAAI 2024 · similarity 0.87)
• FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.86)