Research map / Large language models / Reasoning and chain of thought
DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DP-LLM, which dynamically assigns per-layer precision (between two adjacent bitwidths) at each decoding step using a cheap runtime estimate of relative quantization error, on top of Any-Precision LLM.
Problem
Runtime adaptation of on-device LLMs under varying latency/accuracy constraints via multi-scale quantization requires configuring an adaptation set; static (uniform or layer-wise) precision assignment ignores that layer quantization sensitivity changes across decoding steps.
Approach
Per-layer candidate precision pair (l, h) with a threshold on the relative error ||ΔWx|| computed per token; the threshold is set from a calibrated quantile and average precisions are fine-tuned with a target-precision regularizer; the error is estimated cheaply via linear regression on ||x|| or random projection (JL lemma), with asynchronous use of the previous residual input.
Key results
- On Llama-3-8B at 3.25-bit (5-bit budget), DP-LLM reaches WikiText2 perplexity 7.35 versus 7.47 for HAWQ-V2 and 7.62 for LLM-MQ, and is best across nearly all target precisions tested.
- Latency overhead of the precision selector is a geomean of 1.45% for Llama-3-8B and 0.81% for Phi-3-Medium relative to static LLM-MQ.
- Per-query effective bitwidth increase at the 99th percentile stays below 3% geomean on Alpaca queries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dynamic layer-wise precision assignment instead of static HAWQ-V2 allocation | DP-LLM on Llama-3-8B · WikiText2, 5-bit memory and target 3.50 bits | perplexity: 7.21 -> 7.00 | Helped |
| use dynamic layer-wise precision assignment instead of static HAWQ-V2 allocation | DP-LLM on Llama-3-8B · C4, 5-bit memory and target 3.50 bits | perplexity: 11.41 -> 11.03 | Helped |
| use dynamic layer-wise precision assignment instead of static precision allocation | DP-LLM on Llama-3-8B · MBPP, target 3.50 bits | pass@1: 47.8 -> 50.4 | Helped |
| use dynamic layer-wise precision assignment instead of static precision allocation | DP-LLM on Phi-3-Medium · MBPP, target 3.50 bits | pass@1: 66.3 -> 65.8 | Hurt |
| use neighboring high and low precisions 4 and 5 rather than 3 and 5 (ablation) | DP-LLM on Llama-3-8B · WikiText2, target 4.5-bit precision | perplexity: 6.55 -> 6.36 | Helped |
| replace approximate and asynchronous error estimation with exact error estimation (ablation) | DP-LLM on Llama-3-8B · WikiText2, target 3.5-bit precision | perplexity: 7.00 -> 6.98 | No clear effect |
| add hybrid linear-regression estimation to random projection estimation (ablation) | DP-LLM on Llama-3-8B · Jetson Orin AGX, effective bitwidth 3.5 | latency overhead: 5.04% -> 3.36% | Helped |
| add asynchronous estimation to hybrid estimation (ablation) | DP-LLM on Llama-3-8B · Jetson Orin AGX, effective bitwidth 3.5 | latency overhead: 3.36% -> 2.30% | Helped |
| switch calibration set from C4 to WikiText2 (ablation) | DP-LLM on Llama-3-8B · WikiText2, target 3.5-bit precision | perplexity: 7.00 -> 6.98 | Helped |
| switch calibration set from C4 to WikiText2 (ablation) | DP-LLM on Llama-3-8B · C4, target 3.5-bit precision | perplexity: 11.03 -> 11.05 | Hurt |
Surprising findings
- Dynamic layer-wise sensitivity measured by oracle perplexity change varies substantially across decoding steps, so static mixed-precision based on average sensitivity underperforms the dynamic oracle.
Limitations
- Meets QoS targets only on a best-effort basis; per-query guarantees are left to future work.
- Only optimizes the decoding phase; prefill uses highest precision and is not affected.
- Relies on a single calibration set and a fixed two-precision candidate set per layer.
Experimental setup
Datasets: C4 WikiText2 Alpaca
Benchmarks: WikiText2 C4 GSM8K MBPP BBH MATH-500
Metrics: perplexity exact match pass@1 math_verify accuracy latency (TPOT)
Baselines: LLM-MQ HAWQ-V2
Builds on: Any-Precision LLM LLM-MQ HAWQ-V2 Johnson-Lindenstrauss lemma gpt-fast
Closest papers by meaning
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.92)
- Progressive Mixed-Precision Decoding for Efficient LLM Inference (ICLR 2025 · similarity 0.91)
- ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs (NeurIPS 2026 · similarity 0.90)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.89)
- Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.89)
- Optimizing Large Language Model Training Using FP4 Quantization (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.