Research map / Large language models / Language, safety and interpretability
Localized Zeroth-Order Prompt Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZOPO, a localized zeroth-order prompt optimizer that runs gradient descent in a continuous embedding space of LLM-generated prompts using an NTK-derived Gaussian process surrogate and uncertainty-informed local exploration.
Problem
Black-box discrete prompt optimization for LLMs is query-inefficient because global optimization spends budget searching for a rare global optimum, while local optima are prevalent and well-performing.
Approach
Empirically motivated shift from global to local optimization: maps prompts to continuous embeddings (via a generator LLM plus embedding model), then does zeroth-order gradient descent with a gradient estimator from an NTK-based derived Gaussian process, plus local queries where predictive uncertainty is high.
Key results
- ZOPO achieves the best accuracy on 18 of 30 instruction induction tasks with performance profile rho(5)=0.87, versus 0.57 for the best baseline INSTINCT.
- ZOPO-GPT (ChatGPT-generated prompts with SBERT embeddings) achieves the best result on 15 of 30 tasks, with rho(5)=0.73.
- On GSM8K, ZOPO's CoT prompt reaches 75.4 accuracy versus 71.8 for the hand-crafted 'Let's think step by step'.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZOPO to optimize the chain-of-thought prompt instead of the hand-crafted prompt | zero-shot arithmetic reasoning prompts · GSM8K | accuracy: 71.797 -> 75.360 | Helped |
| Use ZOPO to optimize the chain-of-thought prompt instead of the hand-crafted prompt | zero-shot arithmetic reasoning prompts · AQUA-RAT | accuracy: 52.362 -> 54.724 | Helped |
| Use ZOPO to optimize the chain-of-thought prompt instead of the hand-crafted prompt | zero-shot arithmetic reasoning prompts · SVAMP | accuracy: 76.25 -> 81.0 | Helped |
| Use ZOPO rather than the strongest baseline | instruction induction · 20 instruction induction tasks | test accuracy | Mixed |
| Generate candidates with ChatGPT rather than Vicuna-13B, using SBERT for both (ablation) | ZOPO · instruction induction | test accuracy | Mixed |
| Use structured last-token, OpenAI, or SBERT embeddings instead of random embeddings (ablation) | ZOPO with fixed Vicuna-13B prompts · instruction induction | test accuracy | Helped |
| Replace the NTK kernel with a Matérn kernel (ablation) | ZOPO · instruction induction tasks | test accuracy | Hurt |
| Remove uncertainty-informed local exploration (ablation) | ZOPO · instruction induction tasks | test accuracy | Hurt |
| Increase the generated prompt candidate set from 500 to 1000 or 2000 (ablation) | ZOPO and ZOPO-GPT · instruction induction | test accuracy | No clear effect |
| Use ChatGPT-generated candidates with SBERT rather than Vicuna-generated candidates with SBERT (ablation) | ZOPO · second_word_letter | test accuracy: 24.3_{\pm6.0} -> 96.7_{\pm2.4} | Helped |
| Use ChatGPT-generated candidates with SBERT rather than Vicuna-generated candidates with SBERT (ablation) | ZOPO · cause_and_effect | test accuracy: 86.7_{\pm7.5} -> 80.0_{\pm14.2} | Hurt |
| Use ZOPO-GPT rather than the best baseline | instruction induction · second_word_letter | test accuracy: 74.7_{\pm20.3} -> 96.7_{\pm2.4} | Helped |
Surprising findings
- Prompts generated by ChatGPT (black-box) generally outperform those from white-box Vicuna-13B, and ChatGPT more often yields well-performing local optima.
- Random projection of embeddings performs noticeably worse than structured embeddings, while the best embedding choice is task-dependent.
Limitations
- Gains on several tasks are small or within large standard errors (e.g., antonyms, sum, letters_list ties); some tasks where ZOPO trails baselines (odd_one_out, second_word_letter).
- Results on Vicuna-based configuration cannot be directly compared to ZOPO-GPT because the last-token embedding is tied to the generation model.
- Hyperparameters such as the soft-prompt grid search are tuned on validation sets, and experiments use only 3 runs.
Experimental setup
Datasets: instruction induction (30 tasks) GSM8K AQUA-RAT SVAMP
Benchmarks: instruction induction (20 and 30 tasks) GSM8K AQUA-RAT SVAMP
Metrics: test accuracy performance profile rho(tau) query efficiency
Baselines: APE InstructZero INSTINCT EvoPrompt
Builds on: LoZO InstructZero INSTINCT APE
Closest papers by meaning
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.91)
- Large Language Models as Optimizers (ICLR 2024 · similarity 0.89)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.88)
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.88)
- Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Models (NeurIPS 2024 · similarity 0.88)
- Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.