Research map / Large language models / Reasoning and chain of thought
DynamicInfer: Runtime-Aware Sparse Offloading for LLMs Inference on a Consumer-Grade GPU
Zhui Zhu, Weichen Zhang, Zhenghan Zhou, Yunhao Liu, Fan Dang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynamicInfer, a runtime-aware sparse offloading system that dynamically partitions neurons for LLM inference on consumer-grade GPUs, improving over static partitioning in PowerInfer.
Problem
LLMs have large memory footprints that are hard to deploy on consumer-grade GPUs; prior offloading plus sparse activation systems such as PowerInfer use static neuron partitioning, causing suboptimal GPU utilization and higher latency.
Approach
Replaces static neuron placement between GPU and CPU with runtime-aware dynamic partitioning combined with sparse offloading, so that the neurons placed on the GPU adapt during inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace static hot/cold neuron partition with DynamicInfer runtime scheduling | PowerInfer · ReluLLaMA-70B on PC-High | generation speed: 5.14 -> 7.88 tokens/s | Improved |
| use DynamicInfer instead of PowerInfer | PowerInfer · ReluLLaMA-13B on PC-High | generation speed: 15.5 -> 19.1 tokens/s | Improved |
| use DynamicInfer instead of PowerInfer | PowerInfer · ReluLLaMA-13B on PC-Low (11GB GPU) | generation speed: 7.49 -> 8.47 tokens/s | Improved |
| add micro-level neuron scheduling (ablation) | PowerInfer · ReluLLaMA-13B / 70B, PC-High | speedup: +11.3% / +27.6% | Improved |
| add macro-level scheduling on top of micro-level (ablation) | PowerInfer + micro · ReluLLaMA-13B / 70B, PC-High | speedup: 11.3% -> 13.6% / 27.6% -> 34.0% | Improved |
| add dynamic threshold on top of micro+macro (ablation) | PowerInfer + micro + macro · ReluLLaMA-13B / 70B, PC-High | speedup: 13.6% -> 23.1% / 34.0% -> 57.1% | Improved |
| move more active neurons to GPU via dynamic scheduling | PowerInfer · PC-High, ReluLLaMA series | GPU share of active neurons: 51% -> 68% | Improved |
| dynamic scheduling reduces CPU FFN burden | PowerInfer · PC-High | CPU share of FFN latency: 72% -> 57% | Improved |
| apply sparse activation prediction (omit predicted-inactive neurons) | ReluLLaMA / ProSparse dense · RTE, PIQA, COPA, Winogrande | accuracy: e.g. COPA 93.00 -> 92.00 (70B) | No clear effect |
| add dynamic scheduling and prediction buffers | PowerInfer · ReluLLaMA-13B on 2080Ti | CPU memory: 21.3 -> 36.6 GB | Hurt |
Experimental setup
Metrics: latency GPU utilization
Baselines: PowerInfer
Builds on: PowerInfer
Closest papers by meaning
- Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.89)
- Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding (NeurIPS 2025 · similarity 0.88)
- Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference (ICML 2024 · similarity 0.88)
- Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.88)
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.88)
- Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
