Research map / Vision-language and multimodal / CLIP and zero-shot
Fair-VPT: Fair Visual Prompt Tuning for Image Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies unfairness in visual prompt tuning caused by biased pre-trained ViT and proposes Fair-VPT, which splits prompts into target and cleaner prompts using masked self-attention and a contrastive disentanglement loss to improve fairness on image classification.
Problem
Visual Prompt Tuning of ViT can yield serious unfairness with respect to sensitive attributes (e.g., gender) in downstream classification, largely because the pre-trained ViT encodes biased information.
Approach
Categorize prompts into cleaner prompts and target prompts, encode the class token both with and without masking target prompts in self-attention, train each token to predict sensitive or target attributes respectively, add a supervised contrastive disentanglement loss, and at inference exclude the cleaner prompts via masking.
Key results
- On CelebA (Attractive), Fair-VPT reduces equalized odds to 12.0 from 32.1 for VPT and raises balanced accuracy to 76.3 from 75.0.
- On UTK Face, Fair-VPT achieves equalized odds 4.9 and balanced accuracy 90.9, versus 12.6 and 89.0 for VPT.
- On bFFHQ, Fair-VPT reaches worst-group accuracy 61.9 and equalized odds 37.1, improving over VPT's 46.3 EO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Fair-VPT instead of VPT+FSCL+ | ViT-B/16 prompt tuning · CelebA Attractive | Equalized Odds: 20.6 -> 12.0 | Helped |
| Apply Fair-VPT instead of VPT+AT | ViT-B/16 prompt tuning · CelebA Big Nose | Equalized Odds: 23.7 -> 15.9 | Helped |
| Apply Fair-VPT instead of VPT+FSCL+ | ViT-B/16 prompt tuning · UTK Face Race | Equalized Odds: 9.9 -> 4.9 | Helped |
| Apply Fair-VPT instead of VPT+AT | ViT-B/16 prompt tuning · bFFHQ Age | Equalized Odds: 43.1 -> 37.1 | Helped |
| Apply Fair-VPT instead of VPT+AT | ViT-B/16 prompt tuning · Waterbirds | Equalized Odds: 27.0 -> 18.7 | Helped |
| Add disentanglement loss (ablation) | Fair-VPT · CelebA Attractive | Equalized Odds: 15.0 -> 12.0 | Helped |
| Add disentanglement loss (ablation) | Fair-VPT · UTK Face Race | Equalized Odds: 8.1 -> 4.9 | Helped |
| Add target classification loss (ablation) | Fair-VPT · CelebA Attractive | Equalized Odds: 25.2 -> 12.0 | Helped |
| Add target classification loss (ablation) | Fair-VPT · UTK Face Race | Equalized Odds: 10.9 -> 4.9 | Helped |
| Use standard MSA rather than masking target prompts for target-token encoding (ablation) | Fair-VPT · CelebA Attractive | Equalized Odds: 29.1 -> 12.0 | Helped |
| Use standard MSA rather than masking target prompts for target-token encoding (ablation) | Fair-VPT · UTK Face Race | Equalized Odds: 12.2 -> 4.9 | Helped |
| Add prompt categorization (ablation) | Fair-VPT · CelebA Attractive | Equalized Odds: 32.1 -> 12.0 | Helped |
Surprising findings
- Plain ViT with a nearest class mean classifier (no training) is the most unfair configuration, suggesting the pre-trained backbone is the main source of unfairness rather than the prompts or head.
Limitations
- Trade-off between fairness and overall accuracy on some attributes (e.g., Attractive accuracy drops relative to VPT).
- Evaluated only on ViT-B/16 pre-trained on ImageNet-21k and on a small set of classification benchmarks.
Experimental setup
Datasets: CelebA UTK Face bFFHQ Waterbirds
Benchmarks: CelebA UTK Face bFFHQ Waterbirds
Metrics: accuracy balanced accuracy equalized odds worst-group accuracy
Baselines: ViT VPT VPT+AT VPT+FSCL+
Builds on: Visual Prompt Tuning Vision Transformer supervised contrastive learning FSCL
Closest papers by meaning
- DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers (CVPR 2025 · similarity 0.91)
- Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning? (ICLR 2024 · similarity 0.91)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.87)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.86)
- Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning (ICLR 2024 · similarity 0.86)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.