Research map / Vision-language and multimodal / CLIP and zero-shot
FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates how SemDeDup semantic deduplication of LAION-400M affects fairness of CLIP models and proposes FairDeDup, a modified selection heuristic that boosts underrepresented sensitive concepts while preserving zero-shot performance.
Problem
Dataset deduplication reduces VLP training cost but its effect on social biases in trained vision-language models is unexamined; existing deduplication may amplify or reduce subgroup disparities.
Approach
Replace SemDeDup's max-distance-from-centroid selection heuristic with one that, within each k-means cluster, keeps the sample most similar to the least-represented natural-language sensitive concept prototype (tracked as running per-cluster averages), so deduplication preserves demographic diversity.
Key results
- FairDeDup (50% data) matches SemDeDup/full data on zero-shot benchmarks within about 0.8% (e.g., IN1K acc@5 .897 vs .899 full).
- On FACET gender disparity, SemDeDup worst-class .346 vs FairDeDup .298 (full data .303).
- On FairFace, FairDeDup reduces gender MinSkew to .182 vs SemDeDup .223 (full data .159), and race MaxSkew to .372 vs full .432.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prune 50% of LAION-400M with SemDeDup | CLIP ViT-B/16 · ImageNet1K | Accuracy@5: .899 -> .897 | No clear effect |
| Prune 50% of LAION-400M with FairDeDup instead of SemDeDup | CLIP ViT-B/16 · ImageNet1K | Accuracy@5: .897 -> .897 | No clear effect |
| Prune 50% of LAION-400M with SemDeDup | CLIP ViT-B/16 · FACET gender (masculine/feminine) | Worst-class disparity: .303 -> .346 | Hurt |
| Use FairDeDup instead of SemDeDup | CLIP ViT-B/16 · FACET gender (masculine/feminine) | Worst-class disparity: .346 -> .298 | Helped |
| Prune 50% of LAION-400M with SemDeDup | CLIP ViT-B/16 · FACET skin tone (light/dark) | Worst-class disparity: .354 -> .342 | Helped |
| Use FairDeDup instead of SemDeDup | CLIP ViT-B/16 · FACET skin tone (light/dark) | Worst-class disparity: .342 -> .320 | Helped |
| Prune 50% of LAION-400M with SemDeDup | CLIP ViT-B/16 · FACET age (middle/younger) | Disparity gap: .205 -> .171 | Helped |
| Use FairDeDup instead of SemDeDup | CLIP ViT-B/16 · FACET age (middle/younger) | Disparity gap: .171 -> .150 | Helped |
| Prune 50% of LAION-400M with SemDeDup | CLIP ViT-B/16 · FairFace gender | NDKL: .010 -> .015 | Hurt |
| Use FairDeDup instead of SemDeDup | CLIP ViT-B/16 · FairFace gender | NDKL: .015 -> .012 | Helped |
| Prune 50% of LAION-400M with SemDeDup | CLIP ViT-B/16 · FairFace race | NDKL: .035 -> .034 | Helped |
| Use FairDeDup instead of SemDeDup | CLIP ViT-B/16 · FairFace race | NDKL: .034 -> .030 | Helped |
Surprising findings
- SemDeDup reduces average and worst-class age disparity on FACET while increasing gender disparity.
- FairDeDup on FairFace race fairness outperforms the full-data baseline in several metrics.
Limitations
- Clustering restricts selection so cross-cluster balancing of sensitive concepts is impossible.
- Deduplication inherits social-construct biases of the embedding model; sensitive attribute labels in fairness datasets are subjective and binary-gender-limited.
- Only text-based concept prototypes and gender, skin tone, age, race attributes are studied.
Experimental setup
Datasets: LAION-400M
Benchmarks: CLIP Benchmark (41 datasets; ImageNet1K, ImageNetV2, CIFAR-10/100, Flickr30k, COCO) FACET FairFace
Metrics: Accuracy@5 Recall@5 disparity in recall (mean, max, gap) MinSkew MaxSkew NDKL
Baselines: SemDeDup Full LAION-400M training
Builds on: SemDeDup CLIP D2 Pruning
Closest papers by meaning
- FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models (NeurIPS 2025 · similarity 0.88)
- Joint Vision-Language Social Bias Removal for CLIP (CVPR 2025 · similarity 0.88)
- FairCLIP: Harnessing Fairness in Vision-Language Learning (CVPR 2024 · similarity 0.88)
- Sieve: Multimodal Dataset Pruning using Image Captioning Models (CVPR 2024 · similarity 0.87)
- A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks (NeurIPS 2024 · similarity 0.87)
- Interpretable Debiasing of Vision-Language Models for Social Fairness (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.