Research map / Architectures and efficiency / Quantization
Progressive Mixed-Precision Decoding for Efficient LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PMPD, a phase-aware mixed-precision scheme for LLM weight quantization that uses higher precision for prefill and progressively lower precision during decoding, with static and learned precision-switching schedulers.
Problem
Post-training weight quantization at extremely low precision (2/3-bit) for LLMs on resource-constrained devices causes severe quality degradation when a uniform precision is used across prefill and decoding.
Approach
Allocate higher weight precision to the prefill phase and progressively lower precision deeper into decoding (since later tokens are more resilient), using a prompt-agnostic static scheduler tuned on a validation set or a task-agnostic learned scheduler that reads prefilled KV cache to choose switch points.
Key results
- PMPD-Static matches the 3-bit baseline on CNN/DM and Dialogsum with up to 33% bitwidth reduction, and outperforms DNS at equal bitwidth by up to 29% BLEU on IWSLT.
- On NPU simulation, PMPD achieves 3.8-8.0x throughput over fp16 and up to 1.54x over uniform quantization.
- On RTX 4090 and A40, PMPD linear-layer speedups reach 1.40-12.20x over fp16 kernels, up to 1.41x over uniform quantization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 3-bit weights for prefill instead of 2-bit while keeping decoding at 2-bit (ablation) | Vicuna-7B · MT-Bench | Rouge-L: 18.3 -> 28.1 | Helped |
| use 3-bit weights for decoding instead of 2-bit, with 2-bit prefill (ablation) | Vicuna-7B · MT-Bench | Rouge-L: 18.3 -> 22.2 | Helped |
| allocate higher precision to prefill rather than decoding (ablation) | Vicuna-7B · MT-Bench | Rouge-L: 28.1 vs. 22.2 | Mixed |
| use PMPD-Static instead of the 3-bit uniform baseline | Vicuna-7B · CNN/DM | Rouge-L: 24.2 -> 24.3 | Helped |
| use PMPD-Static instead of the 3-bit uniform baseline | Vicuna-7B · Dialogsum | Rouge-L: 24.4 -> 25.0 | Helped |
| use PMPD-Static instead of the 3-bit uniform baseline | Vicuna-7B · IWSLT | BLEU: 31.6 -> 31.0 | Hurt |
| use PMPD-Learned instead of DNS at the same average bitwidth | Vicuna-7B · MT-Bench | Rouge-L: 36.7 -> 39.6 | Helped |
| use PMPD-Learned instead of the 3-bit baseline | Vicuna-7B · MT-Bench | BERTScore: 89.9 -> 88.9 | Hurt |
| use PMPD instead of fp16 for linear-layer inference | LLMs on Nvidia GPUs · LLM linear layers | speedup: 1.4$-$12.2$\times$ | Helped |
| use the last attention block's KV cache as learned-scheduler input (ablation) | Learned scheduler · CNN/DM, IWSLT, and MT-Bench on MobileLLaMA | normalized score | Helped |
| replace the learned scheduler with a random uniform scheduler (ablation) | Learned scheduler · CNN/DM, IWSLT, and MT-Bench on MobileLLaMA | normalized score | Hurt |
| replace the learned scheduler with a random prior scheduler (ablation) | Learned scheduler · CNN/DM, IWSLT, and MT-Bench on MobileLLaMA | normalized score | Hurt |
Surprising findings
- Using higher precision during decoding yields less gain than using higher precision during prefill; a 3-bit prefill with 2-bit decoding substantially outperforms 2-bit prefill with 3-bit decoding on MT-Bench (28.1 vs 22.2 Rouge-L).
- Some PMPD configurations slightly outperform the high-precision baseline on certain tasks (up to 2.4%).
Limitations
- NPU results come from an analytical performance model/simulation rather than real hardware.
- Learned scheduler may suffer from data distribution shift between its training set and test tasks; Static scheduler requires a task-representative validation set.
- GPU speedups for smaller layers are limited by CPU-side kernel launch overhead and memory-bound matrix-vector multiplication.
Experimental setup
Datasets: C4 CNN/DM Dialogsum IWSLT French-English MT-Bench
Benchmarks: CNN/DM Dialogsum IWSLT MT-Bench
Metrics: Rouge-L BERTScore BLEU SacreBLEU throughput kernel latency speedup
Baselines: Baseline-l (low-precision uniform) Baseline-h (high-precision uniform) DNS fp16
Builds on: Any-Precision LLM Dense-and-Sparse Quantization FlightLLM GPTQ
Closest papers by meaning
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.91)
- DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment (NeurIPS 2025 · similarity 0.91)
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.91)
- SqueezeLLM: Dense-and-Sparse Quantization (ICML 2024 · similarity 0.91)
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (ICML 2024 · similarity 0.91)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.