Research map / Large language models / Reasoning and chain of thought

Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs

ICML 2024 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes any-precision LLMs, obtained by incrementally upscaling a 3-bit SqueezeLLM seed model into a single parent model whose bit-prefixes serve 3–8-bit models, plus a bitplane-based GPU kernel that reduces memory traffic at lower bit-widths.

Problem

Deploying multiple LLMs of different sizes multiplies memory cost and training cost, and prior any-precision DNN methods require QAT from scratch and do not reduce memory bandwidth for LLM inference.

Approach

Incremental upscaling: starting from a low-bit non-uniform (clustering-based) PTQ seed model, each cluster is split into two sub-clusters per added bit, so lower-bit models are bit-prefixes of the parent; paired with a bitplane weight layout and GPU kernel that loads only the required bitplanes.

Key results

• Incremental upscaling from a 3-bit SqueezeLLM seed yields 4–8-bit models whose perplexity is nearly identical to independently quantized SqueezeLLM models (increase <0.1 in most cases on Llama-2-7B).
• Serving {3,4,6,8} bits for Llama-2-7B needs 7.9 GB versus 19.1 GB for separate deployment (2.41x memory savings).
• The kernel achieves up to 3.99x speedup over cuBLAS FP16 at 3-bit on RTX 4090 for a (1,4096)x(4096,4096) matvec, matching or beating the SqueezeLLM kernel on desktop and laptop GPUs and outperforming it on Jetson AGX Orin.

Measured effects reported in the paper

Surprising findings

• Incremental upscaling applied to GPTQ and AWQ (uniform methods) fails, with perplexities exploding (e.g., INF) at 4-bit, motivating the use of non-uniform quantization.
• Seed-model quantization plus upscaling takes under a minute (51.8 s for Llama-2-7B) on a CPU without any training.

Limitations

• Custom kernel does not use tensor cores, so it becomes slower than cuBLAS FP16 for large batch sizes such as prefill; the engine falls back to separate dequantization for M above about 16.
• Upscaled models at 4-bit can lose accuracy on some tasks (e.g., OPT-6.7B WinoGrande -2.3 pp change in some settings) and the method is evaluated only on 1.3B–7B models.
• Uniform quantization methods (GPTQ, AWQ) are incompatible with incremental upscaling, restricting the choice of backbone to non-uniform quantization.

Experimental setup

Datasets: WikiText2 C4 Penn Treebank

Benchmarks: WikiText2 PTB C4 ARC-easy ARC-challenge HellaSwag PIQA WinoGrande

Metrics: perplexity zero-shot accuracy matrix-vector latency end-to-end throughput memory footprint

Baselines: SqueezeLLM AWQ GPTQ LUT-GEMM ExLlamaV2 TensorRT-LLM cuBLAS FP16

Builds on: SqueezeLLM any-precision DNN AWQ GPTQ LUT-GEMM

Closest papers by meaning

• DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment (NeurIPS 2025 · similarity 0.92)
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.92)
• Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models (NeurIPS 2024 · similarity 0.91)
• Optimizing Large Language Model Training Using FP4 Quantization (ICML 2025 · similarity 0.91)
• SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models (ICML 2025 · similarity 0.91)
• Treasures in Discarded Weights for LLM Quantization (AAAI 2025 · similarity 0.91)