Research map / Architectures and efficiency / Quantization
Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Q-Palette, a suite of fractional-bit Gaussian quantizers (TCQ, half-TCQ, VQ, NUQ) with CUDA kernels, plus fusion-aware mixed-scheme quantization for data-free weight-only PTQ of LLMs.
Problem
Weight-only post-training quantization of LLMs for memory-bound small-batch inference; existing sophisticated Gaussian quantizers (e.g., TCQ/QTIP) only support a few integer bitwidths and batch size 1, and bit allocation across layers under a memory or latency budget is suboptimal.
Approach
Derives an information-theoretically optimal fractional bit allocation for Gaussianized weights, builds a quantizer collection covering fractional bitwidths (including half-TCQ mixing two TCQ bitwidths) with batch-size-up-to-8 CUDA kernels, and jointly optimizes quantizer choice and layer fusion via an ILP.
Key results
- Data-free Ours-MSQ-Mem on LLaMA 3.1-8B at 3 bits reaches WikiText2 6.28 vs 6.81 for data-free QTIP at 3 bits.
- Ours-TCQ-3.25 on LLaMA 3.2-1B achieves WikiText2 11.30 vs 17.73 for NF and 12.19 for HIGGS at 3.25 bits.
- On RTX 4090 with batch size 1, Ours-TCQ-TC at 2.0 bits gives 3.57x decoding speedup over FP16 vs 2.91x for QTIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Q-Palette TCQ quantizers in memory-constrained MSQ instead of data-free QTIP | LLaMA 3.1-8B · Data-free, 3.00 bits | WikiText2 perplexity: 6.81 -> 6.28 | Helped |
| Use memory-constrained MSQ with Q-Palette instead of HIGGS-MSQ | LLaMA 3.1-8B · Data-free, 3.25 bits | WikiText2 perplexity: 6.39 -> 6.10 | Helped |
| Use Q-Palette TCQ instead of QTIP | LLaMA 2-7B · Data-aware, 2.00 bits | WikiText2 perplexity: 6.84 -> 6.47 | Helped |
| Use optimized Q-Palette kernels instead of QTIP | LLaMA 2-7B · Data-aware, 2.00 bits, batch size 8 | throughput (Toks/s): 386 -> 1684 | Helped |
| Use Q-Palette TCQ-Tensor Core kernels instead of QTIP | LLaMA 3.1-8B · RTX 4090, batch size 1, 2.0 bits | decoding-latency speedup vs FP16: 2.91$\times$ to 3.57$\times$ | Helped |
| Increase TCQ quantizer support from integer-only TCQ-2,3,4 to all fractional TCQ bitwidths (ablation) | LLaMA 3.1-8B · 3.25-bit budget | total optimality gap: 0.0327 -> 0.0168 | Helped |
| Replace TCQ with VQ in the quantizer pool (ablation) | LLaMA 3.1-8B · 3.25-bit budget | distortion gap: 0.0198 -> 0.0586 | Hurt |
| Use TCQ-ALL instead of TCQ-2,3,4 (ablation) | LLaMA 3.1-8B · 2.50-bit budget | bit allocation gap: 0.0178 -> 0.0034 | Helped |
| Use half-TCQ to mix adjacent TCQ bitwidths within one layer (ablation) | Q-Palette · Fractional-bit quantization | quantization error | No clear effect |
| Add layer fusion to mixed-scheme quantization | Fusion-aware MSQ · LLaMA 3.1-8B latency-constrained inference | decoding speed: 36% speedup | Helped |
| Reduce online rotations by rotating weights only along the input dimension and sharing rotations across layers with identical inputs | Q-Palette inference · Per Transformer block | online rotations: 14 to 4 | Helped |
| Use Q-Palette NUQ CUDA Core kernels instead of Tensor Core kernels (ablation) | LLaMA 3.1-8B · RTX 4090, batch size 1, 2.0 bits | decoding-latency speedup vs FP16: 3.64$\times$ -> 3.70$\times$ | Helped |
Surprising findings
- TCQ quantizers remain efficient at batch size 8, challenging the belief that sophisticated quantizers are computationally prohibitive beyond batch size 1.
- At batch size 8 on RTX 4090, QTIP achieves only 0.74x (2-bit) speedup versus FP16, whereas Q-Palette kernels reach 3.16x.
Limitations
- Data-aware experiments are limited to LLaMA 2-7B and 13B and only compare against QTIP.
- TCQ kernels incur slightly higher latency than NUQ and VQ kernels.
- The optimality analysis assumes expected distortion replaces actual per-layer error, justified empirically only on random Gaussian matrices.
Experimental setup
Datasets: WikiText2 ARC-easy ARC-challenge HellaSwag PiQA WinoGrande random Gaussian samples (codebook construction)
Benchmarks: WikiText2 ARC-easy ARC-challenge HellaSwag PiQA WinoGrande
Metrics: perplexity zero-shot accuracy decoding throughput decoding latency speedup
Baselines: HQQ NormalFloat HIGGS HIGGS-MSQ QTIP HIGGS-Single
Builds on: QTIP HIGGS linearity theorem Any-Precision LLM FLUTE
Closest papers by meaning
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.91)
- ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization (NeurIPS 2025 · similarity 0.91)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.91)
- GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (NeurIPS 2026 · similarity 0.90)
- QuEST: Stable Training of LLMs with 1-Bit Weights and Activations (ICML 2025 · similarity 0.90)
- ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.