Research map / Optimization / Optimizers
Leveraging First and Zeroth-Order Gradient to Address Imbalanced Black-Box Prompt Tuning via Minimax Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BPT-FZG, a black-box prompt tuning method that maximizes AUC via a pointwise nonconvex-concave saddle-point formulation, updating the low-dimensional prompt latent with zeroth-order gradients and the saddle variables with first-order gradients.
Problem
Black-box prompt tuning of LLMs degrades on imbalanced downstream classification data, where cross-entropy leads to collapse on minority classes.
Approach
Replace pairwise AUC maximization with an equivalent pointwise stochastic min-max (saddle point) problem, optimize the prompt latent z in a random low-dimensional subspace using zeroth-order gradient estimates while computing gradients of the auxiliary variables (a, b, alpha) by backpropagation, and run SGDA with a convergence guarantee.
Key results
- On constructed MRPC at ratio 20, BPT-FZG reaches AUC 0.6125 with RoBERTa-large versus 0.5343 for BBT.
- On Amazon Electronics with RoBERTa-large (prompt length 20), BPT-FZG achieves AUC 0.9317 versus 0.8542 for BBT and 0.9273 for BDPL.
- The paper reports a convergence guarantee to a stationary point of the Moreau envelope for the hybrid first/zeroth-order SGDA method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BPT-FZG instead of baseline prompt-tuning methods | Black-box prompt tuning on CoLA, MRPC, and QQP imbalanced scenarios · CoLA, MRPC, and QQP (τ = 20 and τ = 50) | AUC | Helped |
| Use BPT-FZG instead of oversampling or reweighting | Prompt tuning on CoLA · CoLA, τ = 20 and τ = 50 | AUC | Helped |
| Use BPT-FZG instead of baseline prompt-tuning methods | Black-box prompt tuning on Amazon books and electronics · Amazon books and electronics, prompt lengths 20 and 50 | AUC | Helped |
Surprising findings
- Several baselines (BDPL, BBT, GAP3) fall below the zero-shot Manual Prompt on some imbalanced settings, showing imbalance can hurt prompt tuning.
Limitations
- Reported standard deviations are large and several improvements are within noise (e.g., Llama3 results on CoLA at tau=20).
- Evaluation uses only three GLUE-style datasets and three backbones with 3 random seeds.
- Theoretical analysis relies on assumptions (smoothness, bounded variance) and bounds that are not validated empirically.
Experimental setup
Datasets: CoLA MRPC QQP Amazon Books reviews Amazon Electronics reviews
Benchmarks: CoLA (imbalanced, constructed) MRPC (imbalanced, constructed) QQP (imbalanced, constructed) Amazon Books Amazon Electronics
Metrics: AUC
Baselines: Manual Prompt BDPL GAP3 BBT oversample reweight
Builds on: BBT SGDA Yuan et al. 2021 deep AUC maximization Ying et al. 2016 saddle point AUC
Closest papers by meaning
- EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion (AAAI 2025 · similarity 0.88)
- DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2024 · similarity 0.87)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.87)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.87)
- ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2025 · similarity 0.86)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.