Research map / Large language models / Instruction and fine-tuning
Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EASE, an ordering-aware, query-efficient black-box exemplar selection method for ICL that uses a neural bandit (NeuralUCB) over MPNet embeddings of ordered exemplar sequences, with optimal-transport pruning, and can jointly optimize instructions.
Problem
Selecting a fixed, ordered set of in-context exemplars for black-box LLMs performs well across all test queries, without per-query retrieval or privacy exposure, while accounting for exemplar ordering and instruction interactions.
Approach
Treat exemplar selection as black-box optimization over ordered exemplar sequences, represent each sequence by a pre-trained embedding (so ordering changes the embedding), train a neural network surrogate, and pick candidates via NeuralUCB, with optimal-transport pre-filtering to make a large candidate pool tractable; the same framework extends to jointly choosing an instruction.
Key results
- EASE achieves the best validation accuracy on 17 of 19 language tasks in the main Instruction Induction table (and 22 of 24 in the full table), versus 9 for the strongest baselines Evo and Best-of-N.
- On the clean LR task, EASE outperforms the strongest baseline by 8.3 points (80.0 vs 71.7 for Cosine... reported as 8.3% absolute improvement).
- On remapped-label tasks, EASE beats baselines by 6.7%-10% absolute accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EASE instead of Best-of-N for exemplar selection | GPT-3.5 Turbo ICL · Instruction Induction: auto_categorization | accuracy: 20.0 ±0.0 -> 30.0 ±0.0 | Helped |
| Use EASE instead of Best-of-N for exemplar selection | GPT-3.5 Turbo ICL · Instruction Induction: object_counting | accuracy: 70.0 ±0.0 -> 73.3 ±1.7 | Helped |
| Use EASE instead of the strongest baseline | GPT-3.5 Turbo ICL · LR, 0% noise | accuracy: +8.3% in absolute accuracy | Helped |
| Use EASE instead of Best-of-N for exemplar selection | GPT-3.5 Turbo ICL · LP-variant, 10% noise | accuracy: 73.3 ±1.7 -> 80.0 ±2.9 | Helped |
| Use EASE instead of Best-of-N for exemplar selection | GPT-3.5 Turbo ICL · AG News Remap, 10% noise | accuracy: 40.0 ±0.0 -> 51.7 ±1.7 | Helped |
| Jointly optimize the instruction with exemplars (ablation) | EASE · Instruction Induction: auto_categorization | accuracy: 30.0 ±0.0 -> 56.7 ±1.7 | Helped |
| Jointly optimize the instruction with exemplars (ablation) | EASE · Instruction Induction: antonyms | accuracy: 90.0 ±0.0 -> 85.0 ±0.0 | Hurt |
| Jointly optimize the instruction with exemplars (ablation) | EASE · LR, 10% noise | accuracy: 73.3 ±1.7 -> 45.0 ±15.0 | Hurt |
| Jointly optimize the instruction with exemplars (ablation) | EASE · Instruction Induction: diff | accuracy: 100.0 ±0.0 -> 100.0 ±0.0 | No clear effect |
| Replace task-specific EASE exemplar selection with joint instruction and exemplar optimization (ablation) | EASE · Instruction Induction tasks overall | accuracy | Mixed |
Surprising findings
- Exemplar selection gains diminish as the model is progressively finetuned on a task (gain drops from about 3-10% to 0), supporting the hypothesis that selection matters less when the LLM already knows the task.
- Pure OT-based subset selection performs poorly on its own, but becomes useful when combined with NeuralUCB.
Limitations
- On-the-fly embedding computation for ordered exemplar sequences is the computational bottleneck.
- Requires a suitable validation set, which may not be available in some scenarios.
- Joint instruction optimization does not uniformly help (e.g., LR with 10% noise dropped by 28.3 points).
Experimental setup
Datasets: Instruction Induction AG News SST5 Linear regression (novel) Pig Latin variant (novel) AG News Remap (novel) SST5 Reverse (novel)
Benchmarks: Instruction Induction LR LP-variant AG News Remap SST5 Reverse
Metrics: accuracy (validation and test) standard error over 3 trials
Baselines: DPP MMD OT subset selection Cosine retrieval BM25 Active Inf Evo Best-of-N
Builds on: NeuralUCB APE Instruction Induction MPNet
Closest papers by meaning
- Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimization (NeurIPS 2024 · similarity 0.90)
- Sample Efficient Demonstration Selection for In-Context Learning (ICML 2025 · similarity 0.89)
- Efficient Prompt Optimization Through the Lens of Best Arm Identification (NeurIPS 2024 · similarity 0.87)
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.86)
- Large Language Models as Optimizers (ICLR 2024 · similarity 0.86)
- Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.