Research map / Architectures and efficiency / Mixture of experts
QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a single-GPU multi-tenant serving system for fine-tuned MoE LLMs that consolidates similar experts across models and partially reconfigures non-expert layers per request, cutting turnaround time versus NVIDIA MIG.
Problem
Serving multiple fine-tuned mixture-of-experts LLMs on one memory-constrained GPU is inefficient: time sharing incurs minutes-long model swaps, and NVIDIA MIG splits PCIe bandwidth and compute, raising latency.
Approach
Similarity-based expert consolidation builds one shared expert map across models by ranking per-(layer, expert) L2 distances and assigning experts round-robin, while the non-expert layers are swapped on the GPU at runtime to match the requested model.
Key results
- On a single A100 serving Mixtral variants, the proposed system reaches average TTFT 1.41 s and turnaround 8.78 s versus 5.86 s and 49.67 s for NVIDIA MIG, roughly 85% lower turnaround per the abstract.
- On MT-Bench the consolidated Mixtral model averages 8.16 versus 8.13 for the instruct model and 7.99 for the averaged merge.
- On SAMSum with four Switch Transformer variants, ROUGE-1 holds at 0.463 for the proposed method versus 0.256 for averaging.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Serve multiple Mixtral models using similarity-based expert consolidation and runtime partial reconfiguration instead of NVIDIA MIG | Proposed multi-model serving system · Average request | Turnaround time (seconds): 49.67 -> 8.78 | Helped |
| Use runtime partial reconfiguration to serve multiple models instead of serving one model | Mixtral-8x7B serving · Average request | TTFT (seconds): 0.89 -> 1.41 | Hurt |
| Use runtime partial reconfiguration to serve multiple models instead of serving one model | Mixtral-8x7B serving · Average request | Turnaround time (seconds): 8.34 -> 8.78 | Hurt |
| Apply similarity-based expert consolidation and runtime partial reconfiguration rather than weight averaging | Mixtral-8x7B · WikiText2 | perplexity: 3.89 -> 3.85 | Helped |
| Apply similarity-based expert consolidation and runtime partial reconfiguration rather than weight averaging | Mixtral-8x7B · C4 | perplexity: 7.38 -> 7.33 | Helped |
| Apply similarity-based expert consolidation and runtime partial reconfiguration rather than weight averaging | Mixtral-8x7B · PTB | perplexity: 13.44 -> 13.90 | Hurt |
| Apply similarity-based expert consolidation and runtime partial reconfiguration rather than weight averaging | Mixtral-8x7B · MT-Bench | Avg score: 7.99 -> 8.16 | Helped |
| Use expert consolidation and runtime partial reconfiguration rather than weight averaging | Mixtral-8x7B · TruthfulQA (5-shot) | accuracy: 73.0% -> 70.6% | Hurt |
| Increase the number of served variants from A & B to A, B, C, and D | Proposed Switch Transformer serving · SAMSum | ROUGE-1: 0.491 -> 0.463 | Mixed |
| Increase the number of merged variants from A & B to A, B, C, and D | Averaging baseline · SAMSum | ROUGE-1: 0.422 -> 0.256 | Hurt |
Surprising findings
- The proposed consolidated model scores higher than the official instruct model on MT-Bench (8.16 vs 8.13) despite sharing experts with the base model.
Limitations
- Evaluated only on two MoE families (Mixtral-8x7B and Switch Transformer Base-8) with a small number of fine-tuned variants.
- Single-batch requests only; other inference optimizations such as Flash Attention are out of scope.
- Quality degradation on PTB perplexity relative to the base model, and per-request non-expert swap adds TTFT overhead.
Experimental setup
Datasets: WikiText2 PTB C4 MT-Bench MMLU HellaSwag TruthfulQA SAMSum
Benchmarks: WikiText2 PTB C4 MT-Bench MMLU HellaSwag TruthfulQA SAMSum
Metrics: perplexity MT-Bench score accuracy ROUGE TTFT turnaround time throughput
Baselines: NVIDIA MIG single model serving Mixtral-8x7B-v0.1-avg Averaging merge (Switch Transformer)
Builds on: Mixtral Switch Transformer model averaging NVIDIA MIG
Closest papers by meaning
- Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (ICML 2025 · similarity 0.90)
- Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.88)
- Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference (ICML 2025 · similarity 0.88)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.87)
- FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training (NeurIPS 2025 · similarity 0.87)
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.