Research map / Vision-language and multimodal / Human and embodied modeling
FairCLIP: Harnessing Fairness in Vision-Language Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Harvard-FairVLMed, a 10,000-sample glaucoma SLO fundus + clinical note dataset with demographic attributes, benchmarks fairness of CLIP and BLIP2, and proposes FairCLIP, a Sinkhorn-distance fairness loss added to CLIP pre-training.
Problem
Fairness of medical vision-language foundation models is unexplored because existing medical VL datasets lack detailed demographic attributes, ground-truth labels, and rich clinical text.
Approach
Builds a fairness-oriented medical VL dataset with ground-truth glaucoma labels and rich clinical notes, and proposes FairCLIP, which adds a Sinkhorn (optimal transport) loss that aligns the overall sample similarity distribution with each demographic group's distribution during CLIP training.
Key results
- FairCLIP (ViT-B/16) reduced race DPD from 15.35 to 6.07 and DEOdds from 15.11 to 10.50 in zero-shot transfer relative to CLIP (ViT-B/16).
- FairCLIP (ViT-B/16) raised female-group AUC from 64.62 to 66.81 on gender zero-shot evaluation.
- Medical pre-training improved ES-AUC on ethnicity by as much as 8.38 points, and BLIP2 with medical pre-training matched CLIP (80.10 vs 80.27 AUC).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the Sinkhorn-distance fairness objective | CLIP ViT-B/16 · Harvard-FairVLMed race, zero-shot | ES-AUC: 61.67 ± 0.63 -> 65.50 ± 2.60 | Helped |
| Add the Sinkhorn-distance fairness objective | CLIP ViT-B/16 · Harvard-FairVLMed gender, zero-shot | ES-AUC: 63.21 ± 0.83 -> 65.39 ± 2.39 | Helped |
| Add the Sinkhorn-distance fairness objective | CLIP ViT-B/16 · Harvard-FairVLMed ethnicity, zero-shot | ES-AUC: 63.09 ± 0.26 -> 63.33 ± 0.25 | Helped |
| Add the Sinkhorn-distance fairness objective | CLIP ViT-B/16 · Harvard-FairVLMed language, zero-shot | ES-AUC: 60.19 ± 3.47 -> 62.31 ± 0.96 | Helped |
| Add language features to vision-only features (ablation) | BLIP2 · Harvard-FairVLMed race | ES-AUC: 73.76 -> 76.70 | Helped |
| Add language features to vision-only features (ablation) | BLIP2 · Harvard-FairVLMed gender | ES-AUC: 75.22 -> 79.15 | Helped |
| Add language features to vision-only features (ablation) | BLIP2 · Harvard-FairVLMed ethnicity | ES-AUC: 77.18 -> 75.54 | Hurt |
| Add language features to vision-only features (ablation) | BLIP2 · Harvard-FairVLMed language | ES-AUC: 69.88 -> 66.35 | Hurt |
Surprising findings
- Adversarial fairness training (CLIP w/ Adv) did not consistently outperform standard CLIP across attributes.
- FairCLIP (ViT-L/14) did not improve DPD/DEOdds on race and ethnicity over CLIP (ViT-L/14), indicating the gains are architecture-dependent.
Limitations
- Fairness gains are inconsistent across architectures; FairCLIP with ViT-L/14 worsens DPD on race and gender relative to CLIP.
- Dataset covers only glaucoma patients from one institution (Massachusetts Eye and Ear) with a limited set of racial groups (Asian, Black, White).
- Clinical notes must be summarized by LLMs (GPT-4, PMC-LLAMA, MED42) to fit CLIP's 77-token limit.
Experimental setup
Datasets: Harvard-FairVLMed CheXpert MIMIC-CXR PadChest
Benchmarks: Harvard-FairVLMed
Metrics: AUC ES-AUC DPD DEOdds Group-wise AUC
Baselines: CLIP BLIP2 CLIP-FT BLIP2-FT CLIP w/ Adv PMC-CLIP
Builds on: CLIP BLIP-2 Sinkhorn distance PMC-CLIP
Closest papers by meaning
- Hyperbolic Relational Prompts for Intersectional Fairness in Medical VLMs (CVPR 2026 · similarity 0.90)
- FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants (CVPR 2026 · similarity 0.90)
- FairSeg: A Large-Scale Medical Image Segmentation Dataset for Fairness Learning Using Segment Anything Model with Fair Error-Bound Scaling (ICLR 2024 · similarity 0.88)
- FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis (ICLR 2024 · similarity 0.88)
- CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models (NeurIPS 2024 · similarity 0.88)
- FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplication (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.