Research map / Architectures and efficiency / Quantization

HCInfer: An Efficient Inference System via Heterogeneous Error Compensation for Resource-Constrained Devices

NeurIPS 2026 · Architectures and efficiency › Quantization · approach: KV cache quantization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HCInfer, a CPU-GPU heterogeneous inference system that keeps a quantized backbone on GPU while offloading LoRA-style quantization-error compensation to CPU with asynchronous pipelining and sensitivity-aware dynamic rank allocation.

Problem

Deploying LLMs on memory-constrained consumer GPUs trades off accuracy (aggressive quantization) against throughput (offloading over PCIe); low-rank error compensation improves accuracy but its parameters do not fit in scarce GPU memory.

Approach

Offload memory-bound low-rank quantization-error compensation to CPU DRAM while the compute-bound quantized backbone runs on GPU, overlap the two asynchronously, and allocate compensation ranks per matrix using singular-value salience, KL-based output sensitivity, and MoE routing weights.

Key results

• On Qwen-30B-A3B (4-bit), HCInfer improves accuracy by +2.3% (ARC-Easy), +4.3% (ARC-Challenge), +5.2% (MathQA) over the best quantized baseline.
• On Llama-3.1-8B (3-bit, 8GB VRAM limit), HCInfer achieves up to 10.4x throughput over full-precision offloading on MathQA and 8.4x/8.6x on ARC-Easy/ARC-Challenge.
• HCInfer's throughput is 2.3x to 2.8x that of offloading schemes on Qwen-30B-A3B, and it improves over naive compensation offloading by 3.7x to 8.0x.

Surprising findings

• Compensation parameters are memory-bound, so offloading them to CPU while sending only activations is faster than loading weights to GPU on demand.

Limitations

• Throughput gains depend on CPU compute and PCIe bandwidth; CPU parallelism limits prompt (prefill) processing.
• Evaluated on only two models and a single consumer hardware setup (RTX 4090 with i9-10980XE).
• Rank allocation relies on a per-matrix KL sensitivity profiling step before deployment.

Experimental setup

Datasets: WikiText2 C4

Benchmarks: ARC-Easy ARC-Challenge MathQA MMLU WikiText2 C4

Metrics: accuracy perplexity throughput (tokens/s) speedup

Baselines: GPTQ INT4 llama.cpp INT4 GPTQ INT3 llama.cpp INT3 vLLM Hugging Face Accelerate full-precision model

Builds on: GPTQ LoRA Hugging Face Transformers

Closest papers by meaning

• Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding (NeurIPS 2025 · similarity 0.87)
• A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints (ICML 2026 · similarity 0.87)
• Prima.cpp: Fast 30-70B LLM Inference on Heterogeneous and Low-Resource Home Clusters (ICLR 2026 · similarity 0.86)
• Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation (NeurIPS 2026 · similarity 0.86)
• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (NeurIPS 2025 · similarity 0.86)
• SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.86)