Research map / Architectures and efficiency / Mixture of experts
PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PT-MoE, a prompt tuning method that combines low-rank decomposition of soft prompts (shared B, per-expert A_i) with a top-k router over experts, evaluated on 12 MRQA QA datasets and math word problems with LLaMA-3.2-1B-Instruct.
Problem
Prompt tuning (PT) and PEFT variants show inconsistent gains: MoE routing and matrix decomposition each reduce performance on some tasks, and no single PEFT method performs consistently across QA and math.
Approach
Decompose each expert soft prompt into an expert-specific matrix A_i and a shared projection B, and route inputs over the experts with a mean-embedding linear router, so that decomposition gives parameter sharing while MoE gives input-dependent adaptation.
Key results
- PT-MoE reaches 58.26 average F1 on MRQA vs 56.77 for PT and 56.25 for SMoP.
- PT-MoE reaches 56.91% average accuracy on math tasks vs 46.16% for PT and 56.47% for LoRA, with 80k parameters vs LoRA's 106k.
- With a 3B base model on math, PT-MoE averages 71.11% vs 66.43% for PT and 69.61% for SMoP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine low-rank prompt decomposition with MoE routing | Prompt tuning · MRQA | F1 (average): 56.77 -> 58.26 | Helped |
| combine low-rank prompt decomposition with MoE routing | Prompt tuning · Mathematical problem-solving tasks | accuracy (average): 46.16 -> 56.91 | Helped |
| replace PT with LoRA | Parameter-efficient fine-tuning · Mathematical problem-solving tasks | accuracy (average): 46.16 -> 56.47 | Helped |
| add MoE routing without decomposition | Prompt tuning (SMoP) · MRQA | F1 (average): 56.77 -> 56.25 | Hurt |
| add matrix decomposition without MoE routing | Prompt tuning (DPT) · MRQA | F1 (average): 56.77 -> 55.77 | Hurt |
| set soft prompt length to 40 tokens (ablation) | PT-MoE · MRQA in-domain | F1: peak F1 60.66% | Helped |
| set soft prompt length to 40 tokens (ablation) | PT-MoE · MRQA out-of-domain | F1: peak F1 55.28% | Helped |
| use one expert instead of the default two (ablation) | PT-MoE · MRQA in-domain | F1: 58.90% | Hurt |
| use one expert instead of the default two (ablation) | PT-MoE · MRQA out-of-domain | F1: 52.64% | Hurt |
| increase trainable parameters from 18K to 163K (ablation) | PT-MoE · MRQA in-domain | F1: 57.51% to 61.04% | Helped |
| increase trainable parameters from 18K to 163K (ablation) | PT-MoE · MRQA out-of-domain | F1: 53.96% to 55.38% | Helped |
| use selective rather than non-selective routing (ablation) | PT-MoE with probationary routing · MRQA in-domain | F1: 59.24% -> 60.66% | Helped |
Surprising findings
- Individually, matrix decomposition (DPT) and MoE (SMoP) both underperform standard prompt tuning on MRQA average F1.
- Mathematical-task gains over PT are large (+10.75 points) while the gain over LoRA is small (+0.44 points), and HydraLoRA slightly underperforms LoRA on math.
Limitations
- Evaluated only on LLaMA-3.2 1B and 3B models and on extractive QA and math tasks.
- Single-run results with small gains on several metrics; no variance reported.
- Reported parameter savings are inconsistent (25% fewer in abstract vs 75% of LoRA's parameters in text).
Experimental setup
Datasets: MRQA (SQuAD, TriviaQA, SearchQA, HotpotQA, NaturalQuestions, NewsQA, BioASQ, DROP, DuoRC, RACE, RelationExtraction, Text GSM8K (via MetaMath) SVAMP ASDIV MAWPS MATH_PROBLEMS
Benchmarks: MRQA (12 datasets) GSM8K SVAMP ASDIV MAWPS MATH_PROBLEMS
Metrics: F1 Exact Match accuracy
Baselines: Full fine-tuning LoRA HydraLoRA PT DPT SMoP ATTEMPT
Builds on: prompt tuning SMoP DPT HydraLoRA ATTEMPT
Closest papers by meaning
- FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts (NeurIPS 2025 · similarity 0.90)
- From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs (NeurIPS 2026 · similarity 0.90)
- TAIA: Large Language Models are Out-of-Distribution Data Learners (NeurIPS 2024 · similarity 0.89)
- CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning (NeurIPS 2024 · similarity 0.88)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.88)
- Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.