Research map / Large language models / Language, safety and interpretability
InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Optimizes a low-dimensional soft prompt for an open-source LLM (Vicuna-13B) with latent-space Bayesian optimization to generate instructions that maximize zero-shot execution accuracy of a black-box LLM (ChatGPT).
Problem
Finding the best discrete instruction for black-box LLMs accessible only via text APIs is a combinatorial, black-box optimization problem that gradient-based methods cannot solve.
Approach
Optimize a continuous soft prompt fed to an open-source LLM (which generates instructions via in-context learning from exemplars) using Bayesian optimization with a random projection to low dimension and an instruction-coupled kernel that aligns soft-prompt similarity with the similarity of the resulting instructions' zero-shot predictions on the black-box LLM.
Key results
- InstructZero achieves the best zero-shot test accuracy on all 32 of 32 tasks with ChatGPT among APE, Uniform, and InstructZero, with improvements of 20% to 100% on most tasks.
- On Letters List and Sum, InstructZero reaches 1.00 accuracy; on Negation it reaches 0.80 versus 0.27 for manual prompt.
- Instruction-coupled kernel outperforms the standard latent-space kernel on most of 10 tested tasks (e.g., Negation 0.80 vs 0.39).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize a soft prompt rather than use a handcrafted meta-prompt (ablation) | InstructZero instruction generation · Cause_and_effect; Negation; Translation EN-FR; Sum; Formality; Letters List; Larger Animal | execution accuracy: Manual: 0.36, 0.27, 0.02, 0.00, 0.59, 0.00, 0.49; InstructZero: 0.91, 0.80, 0.89, 1.00, 0.63, 1.00, 0.91 | Helped |
| Optimize a soft prompt rather than generate instructions from exemplars only (ablation) | InstructZero instruction generation · Cause_and_effect; Negation; Translation EN-FR; Sum; Formality; Letters List; Larger Animal | execution accuracy: w/o Manual: 0.56, 0.01, 0.47, 0.00, 0.31, 0.15, 0.81; InstructZero: 0.91, 0.80, 0.89, 1.00, 0.63, 1.00, 0.91 | Helped |
| Add instruction similarity to the latent-space kernel (ablation) | InstructZero · Sentiment; Negation; Larger Animal; Second Letter; Formality; Debugging; Odd one out; Ascii; CS algorithm | performance: 0.83 -> 0.93; 0.39 -> 0.80; 0.81 -> 0.91; 0.33 -> 0.62; 0.44 -> 0.63; 0.25 -> 0.50; 0.9 -> 0.92; 0.13 -> 0.33; 0.26 -> 0 | Helped |
| Add instruction similarity to the latent-space kernel (ablation) | InstructZero · Unscrambling | performance: 0.67 -> 0.58 | Hurt |
| Use Bayesian optimization to iteratively select soft prompts | InstructZero · 32 BIG-Bench tasks | accuracy | Helped |
| Use InstructZero instead of APE or Uniform | ChatGPT zero-shot instruction following · 32 BIG-Bench tasks | accuracy | Helped |
Surprising findings
- A 13B open-source Vicuna used as instruction generator, with BO over soft prompts, outperforms APE using the much larger ChatGPT for instruction generation.
- Standard kernel beats instruction-coupled kernel on Unscrambling (0.67 vs 0.58).
Limitations
- Only Vicuna-13B was tried as the open-source LLM.
- Evaluated only on relatively simple instruction-induction tasks; does not cover multi-step planning or human-interactive tasks.
Experimental setup
Datasets: BIG-Bench instruction induction tasks (24 from APE + 8 new)
Benchmarks: BIG-Bench instruction induction (32 tasks)
Metrics: execution accuracy exact match F1
Baselines: APE Uniform sampling BBT BBTv2 Standard kernel (ablation)
Builds on: APE BBT BBTv2 LADDER CMA-ES
Closest papers by meaning
- Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers (ICML 2024 · similarity 0.92)
- WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex Instructions (ICLR 2024 · similarity 0.92)
- PRESTO: Preimage-Informed Instruction Optimization for Prompting Black-Box LLMs (NeurIPS 2025 · similarity 0.91)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.90)
- Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions (ICLR 2024 · similarity 0.90)
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.