AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

DynamicInfer: Runtime-Aware Sparse Offloading for LLMs Inference on a Consumer-Grade GPU

Zhui Zhu, Weichen Zhang, Zhenghan Zhou, Yunhao Liu, Fan Dang

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DynamicInfer, a runtime-aware sparse offloading system that dynamically partitions neurons for LLM inference on consumer-grade GPUs, improving over static partitioning in PowerInfer.

Problem

LLMs have large memory footprints that are hard to deploy on consumer-grade GPUs; prior offloading plus sparse activation systems such as PowerInfer use static neuron partitioning, causing suboptimal GPU utilization and higher latency.

Approach

Replaces static neuron placement between GPU and CPU with runtime-aware dynamic partitioning combined with sparse offloading, so that the neurons placed on the GPU adapt during inference.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace static hot/cold neuron partition with DynamicInfer runtime schedulingPowerInfer · ReluLLaMA-70B on PC-Highgeneration speed: 5.14 -> 7.88 tokens/sImproved
use DynamicInfer instead of PowerInferPowerInfer · ReluLLaMA-13B on PC-Highgeneration speed: 15.5 -> 19.1 tokens/sImproved
use DynamicInfer instead of PowerInferPowerInfer · ReluLLaMA-13B on PC-Low (11GB GPU)generation speed: 7.49 -> 8.47 tokens/sImproved
add micro-level neuron scheduling (ablation)PowerInfer · ReluLLaMA-13B / 70B, PC-Highspeedup: +11.3% / +27.6%Improved
add macro-level scheduling on top of micro-level (ablation)PowerInfer + micro · ReluLLaMA-13B / 70B, PC-Highspeedup: 11.3% -> 13.6% / 27.6% -> 34.0%Improved
add dynamic threshold on top of micro+macro (ablation)PowerInfer + micro + macro · ReluLLaMA-13B / 70B, PC-Highspeedup: 13.6% -> 23.1% / 34.0% -> 57.1%Improved
move more active neurons to GPU via dynamic schedulingPowerInfer · PC-High, ReluLLaMA seriesGPU share of active neurons: 51% -> 68%Improved
dynamic scheduling reduces CPU FFN burdenPowerInfer · PC-HighCPU share of FFN latency: 72% -> 57%Improved
apply sparse activation prediction (omit predicted-inactive neurons)ReluLLaMA / ProSparse dense · RTE, PIQA, COPA, Winograndeaccuracy: e.g. COPA 93.00 -> 92.00 (70B)No clear effect
add dynamic scheduling and prediction buffersPowerInfer · ReluLLaMA-13B on 2080TiCPU memory: 21.3 -> 36.6 GBHurt

Experimental setup

Metrics: latency GPU utilization

Baselines: PowerInfer

Builds on: PowerInfer

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy