Research map / Vision-language and multimodal / Human and embodied modeling
SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SANER, an annotation-free debiasing layer for CLIP text features that neutralizes attribute words in captions and enforces equidistance to attribute-specific versions, preserving attribute information in explicit descriptions.
Problem
CLIP encodes societal bias (gender, age, race); existing debiasing methods either erase explicitly stated attribute information and/or require protected-attribute annotations, limiting dataset choice and downstream utility.
Approach
Attribute-neutralize captions, generate attribute-specific counterparts by word substitution, and train an MLP residual debiasing layer on CLIP text features so the neutral feature is equidistant (low std of cosine similarity) to all attribute-specific features, with reconstruction and contrastive regularization; no attribute labels needed.
Key results
- SANER reduces gender MaxSkew@1000 on FairFace adjective/occupation/activity from 22.9/33.7/19.5 (original) to 8.9/14.5/7.7, beating prompt tuning and projection baselines.
- On Stable Diffusion, SANER keeps gender-specified prompt accuracy at 1.00 while projection-based debiasing drops to 0.58 (female) and 0.79 (male); SP improves from 0.51 to 0.39.
- ImageNet zero-shot accuracy is 65.2% for SANER vs 65.4% original and 64.1% for prompt tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SANER debiasing | CLIP ViT-B/16 · FairFace gender, activity | MaxSkew@1000 (×100): 19.5 to 7.7 | Helped |
| apply SANER debiasing | CLIP ViT-B/16 · FairFace age, adjective | MaxSkew@1000 (×100): 111.1 to 96.0 | Helped |
| apply SANER debiasing | CLIP ViT-B/16 · FairFace race, occupation | MaxSkew@1000 (×100): 57.4 to 45.7 | Helped |
| apply SANER debiasing | Stable Diffusion v2.1 · gender-neutral occupation prompts | statistical parity (SP): 0.51 to 0.39 | Helped |
| apply SANER instead of projection debiasing | Stable Diffusion v2.1 · female-specific prompts | gender accuracy: 0.58 to 1.00 | Helped |
| remove reconstruction and contrastive losses (ablation) | SANER · ImageNet-1K | accuracy (%): 65.4 to 58.8 | Hurt |
| add reconstruction and contrastive losses (ablation) | SANER without regularization · FairFace gender, adjective | MaxSkew@1000 (×100): 15.7 to 8.9 | Helped |
| use contrastive loss only instead of no regularization (ablation) | SANER · FairFace gender, activity | MaxSkew@1000 (×100): 15.3 to 21.7 | Hurt |
| use 100% rather than 50% of COCO training samples (ablation) | SANER · FairFace gender, adjective | MaxSkew@1000 (×100): 15.7 to 8.9 | Helped |
| use reconstruction loss only rather than both regularization losses (ablation) | SANER · FairFace gender, adjective | MaxSkew@1000 (×100): 10.2 to 8.9 | Hurt |
Surprising findings
- Adding only the contrastive loss increases MaxSkew on FairFace relative to no regularization, indicating reconstruction loss is essential.
- Projection-based debiasing increases racial occupation bias on FairFace (57.4 to 75.3).
Limitations
- Bias is not fully eliminated (MaxSkew nonzero); image encoder is not debiased.
- Relies on predefined person-related, gender, age, and race word lists; only binary gender evaluated in depth.
- Several baselines (ARL, Mapper) could not be reproduced, limiting comparison.
Experimental setup
Datasets: COCO FairFace PATA FACET ImageNet-1K
Benchmarks: FairFace PATA FACET ImageNet-1K COCO Karpathy test set
Metrics: MaxSkew@1000 zero-shot accuracy statistical parity accuracy of gender retention FID CLIPScore
Baselines: Original CLIP Prompt tuning-based debiasing Projection-based debiasing Additive Residual Learner (not reproduced) Mapper (not reproduced)
Builds on: CLIP prompt tuning-based debiasing Additive Residual Learner projection-based debiasing MaxSkew
Closest papers by meaning
- Joint Vision-Language Social Bias Removal for CLIP (CVPR 2025 · similarity 0.90)
- Interpretable Debiasing of Vision-Language Models for Social Fairness (CVPR 2026 · similarity 0.88)
- A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks (NeurIPS 2024 · similarity 0.87)
- FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.86)
- Discovering and Mitigating Visual Biases through Keyword Explanation (CVPR 2024 · similarity 0.85)
- MABR: Multilayer Adversarial Bias Removal Without Prior Bias Knowledge (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.