Research map / Architectures and efficiency / Mixture of experts

MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenario

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: Efficient MoE language model systems

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Repurposes speculative decoding's multi-token verification as a lookahead signal for CPU-GPU expert offloading in MoE edge inference, combining a utility estimator, an online integer threshold balancer, and an async prefetch/evict engine.

Problem

MoE inference on memory-constrained edge devices suffers I/O bottlenecks from offloading experts, and autoregressive decoding gives only binary, low-information expert activation signals for prefetching and caching.

Approach

Uses speculative decoding's verification pass to obtain frequency-valued expert activation counts, discretizes them into inertial utility levels, and uses one utility metric to jointly drive CPU-GPU workload partitioning, prefetching and eviction.

Key results

• MoE-SpAc achieves average TPS of 25.06 across seven benchmarks, about 4.04x over the average of standard baselines and 41.9% relative TPS improvement over llama.cpp-w/ SD on average.
• On MMLU-Pro, MoE-SpAc reaches 25.80 TPS versus 17.29 for llama.cpp-w/ SD and 12.22 for HybriMoE on a single RTX 4090.
• On DeepSeek-V2-Lite, MoE-SpAc averages roughly 52.9% TPS improvement over the best baselines on three benchmarks.

Surprising findings

• Reducing expert cache ratio to 17% while spending about 8% of VRAM on a draft model still outperforms baselines with 25% cache, suggesting SD's expert reuse outweighs extra expert cache.

Limitations

• Evaluated only on a single consumer GPU (RTX 4090) with batch size 1, so generalization to other edge hardware is unverified.
• Key theoretical results (convexity, SNR and fault-tolerance claims) rely on simplifying assumptions such as weak inter-token correlation and Binomial activation model.
• Draft model consumes around 8% of VRAM, restricting expert cache capacity and causing OOM at 21% cache ratio.

Experimental setup

Datasets: MMLU-Pro MT-bench HumanEval GSM8K Alpaca CNN/DailyMail QA

Benchmarks: MMLU-Pro MT-bench GSM8K HumanEval Alpaca CNN/DailyMail QA

Metrics: tokens per second latency

Baselines: Accelerate vLLM llama.cpp llama.cpp-w/ SD Mixtral Offloading MoE-Infinity SP-MoE Fate HybriMoE

Builds on: speculative decoding HybriMoE llama.cpp

Closest papers by meaning

• MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (NeurIPS 2025 · similarity 0.89)
• MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.87)
• Mixture of Lookup Experts (ICML 2025 · similarity 0.87)
• Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing (NeurIPS 2026 · similarity 0.86)
• FloE: On-the-Fly MoE Inference on Memory-constrained GPU (ICML 2025 · similarity 0.86)
• Speculative Speculative Decoding (ICLR 2026 · similarity 0.86)