Research map / Architectures and efficiency / Mixture of experts
FloE: On-the-Fly MoE Inference on Memory-constrained GPU
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FloE, an on-the-fly MoE inference system for memory-constrained GPUs that combines contextual sparsification of gate/down projections, INT2 quantization of up projections, cheap inter- and intra-expert sparsity predictors, and a sparse GEMV kernel with compact asynchronous PCIe transfer.
Problem
Running MoE models such as Mixtral-8x7B on memory-constrained consumer GPUs with expert offloading is I/O-bound because activated expert transfers over PCIe cannot be hidden behind computation, hurting single-batch latency.
Approach
Exploit intra-expert activation sparsity (magnitude-based sparsification driven by up-projection outputs) together with ultra-low-bit quantization of only the up projection, and predict next-layer expert and sparsity patterns from current hidden states so transfers can be pipelined with computation.
Key results
- FloE achieves 48.7x end-to-end speedup over DeepSpeed-MII and 2.60x over Mixtral-Offloading on a single RTX 3090, reaching about 91% of Mixtral-GPU speed.
- FloE achieves 9.3x per-expert parameter compression on Mixtral-8x7B and runs on a GPU with 11GB VRAM.
- With only 4.4%-7.6% average performance degradation, and a 9.8% gain over SOTA sparsity methods at 90% sparsity on downstream tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase sparse GEMV sparsity from 0% to 50% (ablation) | FloE sparse GEMV on RTX 3090 · Single-expert execution | latency: 0.542 -> 0.379 ms | Helped |
| Increase sparse GEMV sparsity from 0% to 70% (ablation) | FloE sparse GEMV on RTX 3090 · Single-expert execution | latency: 0.542 -> 0.316 ms | Helped |
| Increase sparse GEMV sparsity from 0% to 90% (ablation) | FloE sparse GEMV on RTX 3090 · Single-expert execution | latency: 0.542 -> 0.283 ms | Helped |
| Prune down-projection inputs rather than up-projection outputs (ablation) | Mixtral-8×7B · WikiText-2 | perplexity: 50% sparsity: perplexity increases by only about 0.5% | Helped |
| Prune SiLU outputs at 70% sparsity (ablation) | Mixtral-8×7B · WikiText-2 | perplexity: above 7 | Hurt |
| Use up-projection activation sparsity instead of the compared SOTA sparsity methods (ablation) | FloE-Wup · Downstream tasks at 90% sparsity | accuracy: 9.8% | Helped |
| Use up-projection activation sparsity at 80% sparsity (ablation) | FloE-Wup · Downstream tasks | accuracy: 2.8% | Helped |
| Combine activation sparsity with quantization (ablation) | FloE · Downstream tasks | accuracy: 4.4% ~ 7.6% average performance degradation | Mixed |
| Use FloE instead of DeepSpeed-MII | Mixtral-8×7B · ShareGPT generation | inference speed: 48.7× | Helped |
| Use FloE instead of Mixtral-Offloading | Mixtral-8×7B · ShareGPT generation | inference speed: 2.60× | Helped |
| Use FloE instead of Fiddler | Mixtral-8×7B · ShareGPT generation | inference speed: 3.14× | Helped |
| Use compact asynchronous transfer instead of PyTorch native transfer (ablation) | MoE expert transfer · DRAM-to-VRAM expert transfer | transfer speed: 12.6× | Helped |
Surprising findings
- Pruning the down-projection inputs is least sensitive to sparsity, yet the up-projection output is chosen for sparsification because it is compatible with predicting sparsity before computation.
- At INT1, the up projection yields only 46.01% of the perplexity of gate-projection quantization, suggesting the up projection is least sensitive to ultra-low-bit quantization.
Limitations
- Accuracy drops noticeably on MMLU relative to the up-projection-only variant, especially under tight memory budgets.
- Evaluation focuses on Mixtral-8x7B for downstream tasks and on a single consumer GPU setup (RTX 3090).
Experimental setup
Datasets: C4 ShareGPT WikiText-2
Benchmarks: WikiText-2 ARC-easy ARC-challenge BoolQ SciQ OpenBookQA Winogrande MMLU
Metrics: perplexity accuracy tokens per second latency bandwidth utilization
Baselines: DeepSpeed-MII Mixtral-Offloading Fiddler Mixtral-GPU CATS CHESS HQQ INT3
Builds on: CATS TEAL HQQ Mixtral-Offloading Fiddler
Closest papers by meaning
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.88)
- FlashMoE: Fast Distributed MoE in a Single Kernel (NeurIPS 2025 · similarity 0.87)
- Mixture of Lookup Experts (ICML 2025 · similarity 0.87)
- MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenario (NeurIPS 2026 · similarity 0.86)
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.86)
- Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.