Research map / Architectures and efficiency / Mixture of experts
Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fiddler is an inference system that runs MoE LLMs under limited GPU memory by splitting expert execution between CPU and GPU, choosing per expert whether to copy weights to GPU or move activations to CPU based on input size.
Problem
Running large Mixture-of-Experts LLMs in resource-constrained settings with limited GPU memory is slow, because offloading expert weights over PCIe is costly and existing CPU-GPU hybrid systems ignore MoE properties and the differing batching behavior of CPUs and GPUs.
Approach
Model CPU latency as linear in the number of input tokens and GPU latency as constant (dominated by weight transfer), then per expert and per layer choose between GPU execution with weight copy, or CPU execution with activation copy, plus popularity-based placement of experts on GPU.
Key results
- Fiddler achieves 1.26x average speedup in single batch end-to-end inference over the best baseline llama.cpp across environments.
- Fiddler achieves 1.30x average speedup in long prefill TTFT (1.07x over DeepSpeed-MII and 1.65x over Mixtral-Offloading).
- Fiddler achieves 11.57x average beam search throughput over llama.cpp.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dynamic CPU-GPU execution planning and popularity-based GPU expert placement | Fiddler · Single-request end-to-end inference, 15 input/output configurations across two environments | tokens per second: 1.26 times faster than llama.cpp | Helped |
| Use dynamic CPU-GPU execution planning and popularity-based GPU expert placement | Fiddler · Long-context prefill | TTFT: 1.07 times | Helped |
| Use dynamic CPU-GPU execution planning and popularity-based GPU expert placement | Fiddler · Long-context prefill | TTFT: 1.65 times | Helped |
| Use dynamic CPU-GPU execution planning and popularity-based GPU expert placement | Fiddler · Beam search | tokens per second: 11.57 times | Helped |
| Use dynamic CPU-GPU execution planning and popularity-based GPU expert placement | Fiddler · Single-request end-to-end inference, ShareGPT, Environment 1 | tokens per second: 1.81 times | Helped |
| Use dynamic CPU-GPU execution planning and popularity-based GPU expert placement | Fiddler · Single-request end-to-end inference, LMSYS-Chat-1M, Environment 1 | tokens per second: 1.56 times | Helped |
| Use dynamic CPU-GPU execution planning | Fiddler · End-to-end inference on Phi-3.5-MoE | tokens per second: 6.5 times | Helped |
| Use dynamic CPU-GPU execution planning and popularity-based GPU expert placement | Fiddler · Across input lengths and two environments | TTFT: 1.13 times | Helped |
| Use dynamic CPU-GPU execution planning and popularity-based GPU expert placement | Fiddler · Across input and output lengths and two environments | ITL: 1.43 times | Helped |
| Place the most popular experts on the GPU instead of random experts | Fiddler expert placement · Mixtral-8x7B, Environment 1 | expected GPU expert hit rate: 25.2% compared to ... 21.9% | Helped |
| Place the most popular experts on the GPU instead of random experts | Fiddler expert placement · Mixtral-8x7B, Environment 2 | expected GPU expert hit rate: 53.0%, ... 48.8% | Helped |
Surprising findings
- Mixtral-8x7B activations after SiLU are rarely near zero, so ReLU-style sparsity exploitation is difficult for this model.
Limitations
- Assumes non-expert layers always fit in GPU memory.
- Evaluated mainly on Mixtral-8x7B with 16-bit precision and two single-GPU environments.
- Latency model assumes CPU latency is linear and GPU latency constant, calibrated at initialization.
Experimental setup
Datasets: ShareGPT LMSYS-Chat-1M
Benchmarks: ShareGPT-based end-to-end latency long prefill TTFT beam search throughput
Metrics: tokens per second time to first token inter-token latency
Baselines: DeepSpeed-MII Mixtral-Offloading llama.cpp
Builds on: Mixtral-Offloading llama.cpp ZeRO-Infinity
Closest papers by meaning
- Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · similarity 0.88)
- $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · similarity 0.87)
- Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference (ICML 2025 · similarity 0.87)
- MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems (NeurIPS 2025 · similarity 0.86)
- QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfiguration (ICML 2025 · similarity 0.86)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.