Research map / Large language models / Agents, code and math
One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mixture-of-Prompts (MoP), which partitions training demos into clusters via K-means in an embedding space, assigns each cluster an expert with a jointly searched instruction, and routes queries to the nearest expert.
Problem
Automatic prompt optimization searches only a single demo-free instruction, which cannot cover the full complex problem space of a task.
Approach
Adopt the mixture-of-experts paradigm for prompts: each expert is a (demo cluster, instruction) pair. Demos are clustered by semantic similarity, motivated by a kernel-regression view of in-context learning; instructions are generated from other clusters' demos and selected on region-routed validation data (region-based joint search).
Key results
- MoP achieves an average performance of 52.73% versus 41.39% for APE+Demos and 39.87% for IZ+Demos on the main benchmarks.
- MoP reaches an average win rate of 81% against six baseline methods across the three benchmarks.
- On OOD splits, MoP scores 68 on Mathdataset versus 9 for APE and 48 for IZ+K-centroids.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace a single-prompt baseline with MoP | Prompt optimization · Three benchmarks (average) | performance: 41.39% -> 52.73% | Helped |
| Use GPT2-Large rather than Ada embeddings (ablation) | MoP · Mathdataset | score: 95 ± 3.3 -> 97 ± 1.3 | Mixed |
| Use GPT2-Large rather than Ada embeddings (ablation) | MoP · Auto categorization | score: 59 ± 0.4 -> 53 ± 0.9 | Hurt |
| Use K-means-Auto rather than random clustering (ablation) | MoP · Auto categorization | score: 36 ± 0.8 -> 59 ± 4.2 | Helped |
| Use K-means-Auto rather than K-means-Balanced (ablation) | MoP · Taxonomy animal | score: 82 ± 2.6 -> 88 ± 6.9 | Helped |
| Use RBJS rather than Independent Search (ablation) | MoP · Taxonomy animal | score: 71 ± 6.3 -> 88 ± 6.9 | Helped |
| Use RBJS rather than Joint Search without region-based evaluation (ablation) | MoP · Taxonomy animal | score: 82 ± 9.0 -> 88 ± 6.9 | Helped |
| Generate prompts using each expert's own demos rather than other experts' demos (ablation) | MoP · Taxonomy animal | score: 88 ± 6.9 -> 72 ± 1.0 | Hurt |
| Use similarity-based routing rather than random routing (ablation) | MoP · Mathdataset | score: 77 ± 2.2 -> 95 ± 1.7 | Helped |
| Use similarity-based routing rather than random routing (ablation) | MoP · Auto categorization | score: 32 ± 2.9 -> 59 ± 4.2 | Helped |
| Use MoP rather than APE | BIG-Bench-Hard · Causal judgement | execution accuracy: 59.93 ± 0.50 -> 60.99 ± 2.19 | Helped |
| Use MoP rather than APE-Nearest-Neighbor | BIG-Bench-Hard · Movie Recommendation | execution accuracy: 87.00 ± 2.45 -> 81.67 ± 2.36 | Hurt |
Surprising findings
- More experts are not necessarily better; there is an optimal number of partitions for the problem space.
- Validation rankings of candidate instructions on the full data are misaligned with rankings on routed local data, so region-based evaluation is needed.
Limitations
- Search cost grows linearly with the number of experts, although it is parallelizable.
- Relies on simple components (K-means, APE for instruction generation, L2 routing) and the paper notes other algorithms could be used.
Experimental setup
Datasets: Instruction Induction Super Natural Instructions BIG-Bench Hard
Benchmarks: Instruction Induction Super Natural Instructions BIG-Bench Hard
Metrics: task score (predefined score functions) win rate mean and std over 3 runs
Baselines: APE APE+Demos APE+K-centroids InstructZero IZ+Demos IZ+K-centroids OPRO APE-Nearest-Neighbor
Builds on: APE InstructZero OPRO Mixture of Experts
Closest papers by meaning
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.91)
- Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following (AAAI 2024 · similarity 0.88)
- From Instance Training to Instruction Learning: Task Adapters Generation from Instructions (NeurIPS 2024 · similarity 0.87)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.87)
- Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective (ICML 2025 · similarity 0.87)
- Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.