Research map / Large language models / Instruction and fine-tuning
Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mixture of Prompts (MoPs), a gated mixture of soft-prompt experts injected in intermediate layers of compressed LLaMA-7B, to handle task and data heterogeneity in centralized and federated prompt tuning.
Problem
Prompt tuning for multiple tasks and heterogeneous data sources suffers from prompt interference (conflicting training signals, client drift), and compressed LLMs add further approximation issues.
Approach
Replace a single shared soft prompt with K groups of prompts (experts) injected at an intermediate layer, with a shallow MLP gate computed from the early-layer embedding of the input that rescales each expert's prompts, initialized with a one-to-one expert-task mapping.
Key results
- In centralized training, MoPs reduce final perplexity versus prompt-tuning baseline by 3% to 30% (e.g., Dolly-15K 75% unstructured pruning 8.25 to 7.24 PPL; 90% unstructured 52.65 to 40.34).
- In federated training, MoPs beat FedPrompt by up to 50% on unstructured pruning (Dolly-15K 90%: 98.13 to 65.25 PPL) and up to 70% on structured 7:8 pruning (143.02 to 84.10).
- With 85% unstructured pruning, MoPs match LoRA and AI^3 (12.4 PPL), while at 90% pruning LoRA (21.4) remains better than MoPs (26.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoPs instead of prompt tuning without gating | Llama-7B · Dolly-15K centralized, 90% unstructured pruning | perplexity: 52.65 -> 40.34 | Helped |
| Use MoPs instead of prompt tuning without gating | Llama-7B · Dolly-15K centralized, 85% unstructured pruning | perplexity: 18.16 -> 15.04 | Helped |
| Use MoPs instead of prompt tuning without gating | Llama-7B · Super-Natural centralized, 90% unstructured pruning | perplexity: 58.47 -> 52.86 | Helped |
| Use MoPs instead of FedPrompt | Llama-7B · Dolly-15K federated, 90% unstructured pruning | perplexity: 98.13 -> 65.25 | Helped |
| Use MoPs instead of FedPrompt | Llama-7B · Super-Natural federated, structured 7:8 pruning | perplexity: 91.64 -> 72.04 | Helped |
| Inject prompts in earlier layers rather than later layers (ablation) | MoPs · Dolly-15K centralized | perplexity | Helped |
| Increase the number of prompts per expert (ablation) | MoPs · Dolly-15K centralized | convergence speed | Helped |
| Combine Int8 quantization with structured 2:4 pruning and MoPs rather than the baseline | Dolly-15K, federated with 10 clients · Int8 + structured 2:4 (50.0%) | perplexity: 14.24 -> 14.51 | Hurt |
| Combine Int8 quantization with structured 4:8 pruning and MoPs rather than the baseline | Dolly-15K, federated with 10 clients · Int8 + structured 4:8 (50.0%) | perplexity: 13.13 -> 13.10 | No clear effect |
| Use MoPs instead of prompt tuning without gating | Dolly-15K, centralized, uncompressed Llama-7B · Uncompressed | perplexity: 3.24 -> 3.09 | Helped |
Surprising findings
- Gains in the federated setting are larger than in the centralized setting, which the authors attribute to the gate reducing client drift under skewed data.
- Under 90% unstructured pruning, MoPs (26.6 PPL) still trails LoRA (21.4), so soft-prompt variants do not always match weight-modifying PEFT.
Limitations
- Gains are measured by perplexity only, with no downstream task accuracy reported.
- Behind LoRA at the highest pruning ratio (90% unstructured), and the choice of injection layer trades off against backpropagation cost.
- Privacy implications of the federated setup are left as an open question.
Experimental setup
Datasets: Databricks Dolly 15k Super-Natural Instructions
Benchmarks: Databricks Dolly 15k Super-Natural Instructions
Metrics: perplexity
Baselines: Prompt tuning baseline FedPrompt LoRA AI^3
Builds on: prompt tuning FedAvg SparseGPT LLM.int8() prompt injection
Closest papers by meaning
- SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.86)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.86)
- Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights (NeurIPS 2025 · similarity 0.85)
- Efficient multi-prompt evaluation of LLMs (NeurIPS 2024 · similarity 0.85)
- Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following (AAAI 2024 · similarity 0.85)
- Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.