Research map / Vision-language and multimodal / CLIP and zero-shot
BendVLM: Test-Time Debiasing of Vision-Language Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bend-VLM, a fine-tuning-free, query-specific nonlinear test-time debiasing method for VLM text embeddings that combines LLM-based local attribute orthogonalization with reference-image distance equalization.
Problem
Debiasing vision-language model embeddings for online, open-set queries without fine-tuning (which causes catastrophic forgetting) and without a fixed set of classes known in advance, since existing fine-tuning-free methods use one global linear debiasing direction.
Approach
Derives a query-specific (local) attribute subspace by prompting an LLM to produce attribute-augmented rephrasings of each query and orthogonalizes the text embedding against it; then uses a reference image dataset with attribute labels to find the closest unit-norm embedding that is equidistant to the top-n attribute-group images, solved analytically.
Key results
- On CelebA with CLIP-ViT-B-P16 HairColor queries, Bend-VLM reaches KL 0.016 and MaxSkew 0.191 versus 0.059 KL for the best baseline, with worst-group AUC 0.772 vs 0.701 for baseline CLIP.
- On UTKFace Stereotype queries with CLIP-ViT-B-P16, gender KL drops from 0.120 (baseline) to 0.008 for Bend-VLM.
- On FLAVA, Bend-VLM achieves stereotype KL 0.040 vs 0.232 for the best compared method (Orth-Cal.).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Bend-VLM instead of baseline CLIP | CLIP-ViT-B-P16 · CelebA HairColor, gender | KL Divergence: 0.140 ± 0.004 -> 0.016 ± 0.002 | Helped |
| Apply Bend-VLM instead of baseline CLIP | CLIP-ViT-B-P16 · CelebA HairColor, gender | Worst Group AUC: 0.701 ± 0.001 -> 0.772 ± 0.003 | Helped |
| Apply Bend-VLM instead of Orth-Proj. | CLIP-ViT-L-P14 · CelebA HairColor, gender | Worst Group AUC: 0.807 ± 0.002 -> 0.802 ± 0.002 | Hurt |
| Apply Bend-VLM instead of baseline CLIP | CLIP-ViT-B-P16 · CelebA Stereotype, gender | KL Divergence: 0.436 ± 0.010 -> 0.014 ± 0.001 | Helped |
| Apply Bend-VLM to caption conditioning embeddings | ClipCap on FairFace captions · FairFace generated captions | Max Disparity: 0.161 -> 0.095 | Helped |
| Remove Step 1 local-subspace orthogonalization (ablation) | Bend-VLM, CLIP-ViT-B-P16 · CelebA HairColor, gender | Worst Group AUC: 0.772 ± 0.003 -> 0.700 ± 0.004 | Hurt |
| Add Step 1 local-subspace orthogonalization (ablation) | Bend-VLM without Step 1, CLIP-ViT-B-P16 · CelebA HairColor, gender | KL Divergence: 0.036 ± 0.015 -> 0.016 ± 0.002 | Helped |
| Remove Step 2 reference-image equalization (ablation) | Bend-VLM, CLIP-ViT-B-P16 · CelebA HairColor, gender | KL Divergence: 0.016 ± 0.002 -> 0.094 ± 0.006 | Hurt |
| Add Step 2 reference-image equalization (ablation) | Bend-VLM without Step 2, CLIP-ViT-B-P16 · CelebA HairColor, gender | Worst Group AUC: 0.772 ± 0.002 -> 0.772 ± 0.003 | No clear effect |
| Debias gender using the full method | Bend-VLM · FairFace HairColor queries evaluated on race | KL Divergence: 0.606 ± 0.043 -> 0.837 ± 0.035 | Hurt |
| Remove Step 1 orthogonalization (ablation) | Bend-VLM · FairFace HairColor queries, gender debiasing evaluated on race | KL Divergence: 0.837 ± 0.035 -> 0.594 ± 0.074 | Helped |
| Remove Step 2 reference-image equalization (ablation) | Bend-VLM · FairFace HairColor queries, gender debiasing evaluated on race | KL Divergence: 0.837 ± 0.035 -> 0.873 ± 0.024 | No clear effect |
Surprising findings
- Bend-VLM step 2 alone does not increase racial bias after gender debiasing, while orthogonalization (step 1) is the main source of the cross-attribute shortcut issue.
- Orthogonal Projection and Orthogonal Calibration sometimes increase KL divergence or MaxSkew above the baseline CLIP for stereotype queries.
Limitations
- Requires a reference dataset with protected attribute labels, which is not always available.
- Relies on a 7B LLM for attribute augmentation, adding computational overhead.
- Evaluation uses only binary gender labels and race/gender labels that are not self-identified.
Experimental setup
Datasets: FairFace CelebA UTKFace
Benchmarks: CelebA HairColor queries Stereotype queries on UTKFace, FairFace, CelebA FairFace captions sentiment
Metrics: KL divergence MaxSkew Worst Group AUC ROC
Baselines: Baseline CLIP Orth-Proj. Orth-Cal. DebiasCLIP
Builds on: orthogonal projection debiasing Orthogonal Calibration DebiasCLIP
Closest papers by meaning
- Joint Vision-Language Social Bias Removal for CLIP (CVPR 2025 · similarity 0.90)
- A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks (NeurIPS 2024 · similarity 0.90)
- Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation (ICML 2026 · similarity 0.88)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.88)
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.88)
- Interpretable Debiasing of Vision-Language Models for Social Fairness (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.