Research map / Architectures and efficiency / Quantization
MPQ: A Message-Passing View of Post-Training Quantization
Yanshuo Chen, chenxi liu, Reza Shirkavand, Heng Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Casts row-wise post-training quantization as a message-passing problem so that weight blocks are not committed to a single codeword before neighboring blocks are considered.
Problem
Existing PTQ solvers (GPTQ, LDLQ, block coordinate descent) commit each weight block to a single codeword before its neighbors are considered, leaving near-optimal alternatives unexplored when compressing LLM weights to low-bit representations without retraining.
Approach
Recasts row-wise PTQ as a message-passing inference problem over weight blocks, so that multiple candidate codewords per block can be maintained and exchanged with neighbors instead of greedily fixing one per block.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace hard-decision greedy solvers with finite-temperature VAMP plus block CD (MPQ) | Llama3-8B 2-bit E8P12 · WikiText-2 | perplexity: 12.76 -> 11.63 (vs LDLQ + CD) | Improved |
| use MPQ instead of OJBKQ at 3-bit scalar | Llama3-8B · WikiText-2 | perplexity: 7.95 -> 7.81 | Improved |
| use MPQ instead of OJBKQ at 4-bit scalar | Llama3-8B · WikiText-2 | perplexity: 6.48 -> 6.46 | Improved |
| add 3 CD passes to greedy GPTQ (ablation) | GPTQ 3-bit · WikiText-2 | perplexity: 8.29 -> 8.43 (Llama3-8B) | No clear effect |
| add 3 CD passes to LDLQ (ablation) | LDLQ 2-bit E8P12 · WikiText-2 | perplexity: 15.01 -> 12.76 | Improved |
| add AWQ scaling before RHT in MPQ (VAMP-only) (ablation) | MPQ 3-bit scalar · WikiText-2 | perplexity: 31.24 -> 13.76 (Llama3.2-1B) | Improved |
| add AWQ scaling before RHT in full MPQ (ablation) | MPQ 3-bit scalar · WikiText-2 | perplexity: 8.09 -> 7.81 (Llama3-8B) | Improved |
| add CD refinement after VAMP (3-bit scalar, 4 VAMP iterations) (ablation) | MPQ 3-bit scalar · WikiText-2 | perplexity: 7.80 -> 7.88 | No clear effect |
| add CD refinement after VAMP at 2-bit E8P12 (16 VAMP iterations) (ablation) | MPQ 2-bit E8P12 · WikiText-2 | perplexity: 17.24 -> 11.63 | Improved |
| increase VAMP iterations from 1 to 16 with 3 CD sweeps (ablation) | MPQ 2-bit E8P12 · WikiText-2 | perplexity: 14.34 -> 11.63 | Improved |
| use MPQ initialization with D4 codebook instead of LDLQ (ablation) | 2-bit D4 codebook + CD3 · WikiText-2 | perplexity: 14.6389 -> 12.9658 | Improved |
| extend CD refinement of the LDLQ baseline to 20 sweeps (ablation) | LDLQ 2-bit E8P12 · WikiText-2 | perplexity: 12.756 -> 12.651 | No clear effect |
Experimental setup
Baselines: GPTQ LDLQ block coordinate descent
Builds on: GPTQ LDLQ block coordinate descent
Closest papers by meaning
- Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization (NeurIPS 2025 · similarity 0.90)
- CBQ: Cross-Block Quantization for Large Language Models (ICLR 2025 · similarity 0.90)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.90)
- ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals (ICML 2025 · similarity 0.89)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.89)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
