Research map / Vision-language and multimodal / Multimodal LLMs

Interpretable Debiasing of Vision-Language Models for Social Fairness

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DeBiasLens, which trains sparse autoencoders on facial image/caption data without social labels, selects gender/age/race "social neurons" and deactivates them at inference to reduce social bias in CLIP, LLaVA, and InternVL2 while keeping general performance.

Problem

VLMs and LVLMs inherit social biases (e.g., skewed demographics in text-to-image retrieval, definitive answers on ambiguous VQA), and existing debiasing methods act on surface-level signals or degrade general capability because neurons are polysemantic.

Approach

Attach a Matryoshka sparse autoencoder to the last layer of a frozen VLM encoder, identify neurons that are consistently active for one demographic group and absent for others, and deactivate them at inference, mixing the SAE reconstruction with the original hidden state via weight alpha.

Key results

• On CLIP ViT-B/16 with text-side DeBiasLens, Max Skew on Adj prompts drops from 21.9 to 7.1 on FairFace, the best among compared methods.
• On InternVL2-8B, DeBiasLens reduces gender disproportion by roughly 40-50% with a 4-10 point average general performance drop.
• On SBBench with InternVL2-8B, gender accuracy rises from 83.83 to 86.68 (rule-based) with FairFace-trained SAE.

Surprising findings

• SAEs trained on the facial-only FairFace dataset yielded better social neurons than more in-distribution synthetic data, and gender-neuron modulation also affected age predictions.
• Modulating age neurons reduced gender bias, which the authors attribute to age-gender correlations in the data.

Limitations

• Relies on the coverage of SAE training data, which may underrepresent subtle or culturally specific bias; FairFace lacks fine-grained labels.
• Assumes social attributes can be disentangled in a small neuron set, which may fail for complex intersectional bias.
• Only neuron-level modulation is studied; interactions with image-text alignment are not examined.

Experimental setup

Datasets: FairFace CelebA CocoGender CocoGender captions Bias in Bios SBBench SB-Syn SB-Syn-Crop ImageNette MME MMMU SEED-Bench-2

Benchmarks: FairFace Max Skew (T2I retrieval) VLAGenderBias SBBench ImageNette MME MMMU SEED-Bench-2 PATA PAIRS

Metrics: Max Skew gender disproportion rate accuracy on bias probing questions general performance accuracy

Baselines: Prompt Projection Bend-VLM SANER MMNeuron Full Fine-Tuning LoRA Fine-Tuning Pruning Prompt Tuning Prompt Engineering

Builds on: Matryoshka sparse autoencoder MMNeuron Bend-VLM SANER

Closest papers by meaning

• Joint Vision-Language Social Bias Removal for CLIP (CVPR 2025 · similarity 0.90)
• The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Language Models (ICLR 2024 · similarity 0.90)
• A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks (NeurIPS 2024 · similarity 0.88)
• SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP (ICLR 2025 · similarity 0.88)
• BendVLM: Test-Time Debiasing of Vision-Language Embeddings (NeurIPS 2024 · similarity 0.88)
• SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples (CVPR 2024 · similarity 0.87)