Research map / Architectures and efficiency / Pruning and efficient tuning
Controllable Prompt Tuning For Balancing Group Distributional Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Controllable Prompt Tuning (CPT), which balances loss across groups via an entropy-maximizing multi-gradient update with a controllable group-weight vector, implemented as prompt tuning on frozen ViT/CLIP backbones to fight spurious correlations.
Problem
Models trained on data with multiple groups or domains suffer severe degradation under distribution shift; worst-group objectives (GroupDRO) sacrifice performance on other groups and overfit, and full fine-tuning is costly when many trade-off settings are needed.
Approach
Instead of optimizing only the worst group, pick a descent direction in the convex hull of per-group gradients that maximizes the entropy of the group loss distribution (balancing losses), with constraints to not hurt well-performing groups; a controlling vector c adjusts the trade-off, and prompt tuning keeps the approach parameter-efficient.</даn>
Key results
- On Waterbirds CPT reaches 93.5% worst-group and 96.3% average accuracy with 94k trainable parameters (0.4% of the model), beating DFR's 92.9% worst-group.
- On CelebA CPT reaches 92.0% worst-group accuracy versus 91.0% for PDE.
- On ISIC CPT achieves 64.9 AUROC, roughly 10 points above DISC (55.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CPT balancing with prompt tuning | ResNet50 · Waterbirds | worst-group accuracy: 86.7_{\pm 0.6} -> 93.5_{\pm 0.4} | Helped |
| apply CPT balancing with prompt tuning | ResNet50 · CelebA | worst-group accuracy: 86.3_{\pm 1.1} -> 92.0_{\pm 0.3} | Helped |
| apply CPT | ResNet50 · MetaShift | worst-group accuracy: 73.5_{\pm 1.4} -> 77.0}_{\pm{0.8} | Helped |
| apply CPT | ResNet50 · ISIC | AUROC: 55.1_{\pm 2.3} -> 64.9}_{\pm 1.2 | Helped |
| add the entropy objective rather than average group gradients (ablation) | CPT on ResNet50 · Waterbirds | worst-group accuracy: 91.9_{\pm 0.3} -> 91.9_{\pm 0.3} | Helped |
| add the entropy objective rather than average group gradients (ablation) | CPT on ViT-B/16 · MetaShift | worst-group accuracy: 74.7_{\pm 2.5} -> 77.0}_{\pm{0.8} | Helped |
| increase prompt length from N=1 to N=16 (ablation) | CPT on ViT-B/16 · Waterbirds | worst-group accuracy: 92.9_{\pm 0.8} -> \mathbf{93.9}_{\pm 0.7} | Helped |
| increase the controlling weight on minority group 3 (ablation) | CPT · CelebA after one epoch | group 3 accuracy: 91.7 -> 95.6 | Helped |
| increase the controlling weight on minority group 3 (ablation) | CPT · CelebA after one epoch | worst-group accuracy: 91.7 -> 91.1 | Hurt |
| increase the controlling weight on minority group 2 (ablation) | CPT · CelebA after one epoch | group 2 accuracy: 92.8 -> 95.2 | Helped |
Surprising findings
- Plain ERM prompt-tuning on CLIP-style backbones gives high average but poor worst-group accuracy, showing prompt tuning alone does not fix spurious correlations.
- Gradient averaging without the entropy objective (CPT w/o entropy) performs noticeably worse than full CPT on MetaShift worst-group accuracy (74.7 vs 77.0).
Limitations
- A separate prompt set must be trained for each predefined controlling vector c, so cost grows with the number of c values.
- Adding a new controlling vector requires retraining a prompt set from scratch.
- Authors suggest hypernetworks as a future fix, which is not implemented.
Experimental setup
Datasets: Waterbirds CelebA MetaShift ISIC
Benchmarks: Waterbirds CelebA MetaShift ISIC
Metrics: worst-group accuracy average accuracy mean accuracy AUROC number of trainable parameters
Baselines: ERM GroupDRO DFR AFR PDE CGD DISC LISA JTT Subsample UW CnC
Builds on: GroupDRO MGDA prompt tuning CLIP
Closest papers by meaning
- Leveraging First and Zeroth-Order Gradient to Address Imbalanced Black-Box Prompt Tuning via Minimax Optimization (AAAI 2025 · similarity 0.85)
- The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations (NeurIPS 2024 · similarity 0.84)
- Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment (AAAI 2025 · similarity 0.84)
- Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency (ICLR 2025 · similarity 0.84)
- MixMax: Distributional Robustness in Function Space via Optimal Data Mixtures (ICLR 2025 · similarity 0.84)
- Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.