Research map / Architectures and efficiency / Quantization
Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GLVQ, a post-training LLM weight quantizer that learns a per-group lattice generation matrix and per-group mu-law companding, using Babai rounding for index assignment, combined with Slim-LLM salience-based bit allocation.
Problem
Post-training quantization of LLM weights to ultra-low bit-widths (2-3 bits) causes large accuracy degradation with uniform or fixed-lattice quantizers, and free-form VQ (AQLM) is slow to decode.
Approach
Assign each weight group its own learnable lattice codebook (generation matrix G_g) plus a learnable group-specific mu-law companding curvature, with integer indices set by Babai rounding so only G_g and mu_g are optimized; decoding is a matrix-vector product.
Key results
- On Llama 2-70B at 2-bit, GLVQ-32D reaches 3.36 Wikitext-2 perplexity vs 3.78 for QTIP and 3.91 for QuIP#.
- At 1.5-bit on Llama-2 7B, GLVQ reaches 7.01 Wikitext-2 perplexity, versus 7.32 for PV-Tuning at 1.58 bits.
- GLVQ-32D decodes at 82.0 TOK/s on Llama 2-7B 2-bit vs 105.2 for QTIP, a throughput cost of about 22%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GLVQ-32D instead of QuIP# | 2-bit Llama 2-70B quantization · WikiText-2 | perplexity: 3.91 -> 3.36 | Helped |
| use GLVQ-8D instead of QuIP# | 2-bit Llama 2-13B quantization · ARC-Challenge | accuracy: 39.5 -> 40.0 | Helped |
| use group-wise salience-driven bit allocation instead of uniform bit-width (ablation) | GLVQ-8D · WikiText-2, Llama 1-13B, 2-bit | perplexity: 5.79 -> 5.64 | Helped |
| learn group-specific lattice bases instead of sharing one fixed basis (ablation) | GLVQ-8D · WikiText-2, Llama 1-13B, 2-bit | perplexity: 5.89 -> 5.64 | Helped |
| use group-specific companding instead of a fixed global transformation (ablation) | GLVQ-8D · WikiText-2, Llama 1-13B, 2-bit | perplexity: 5.87 -> 5.64 | Helped |
| reduce group size from 128 to 32 (ablation) | GLVQ · WikiText2, Llama 1-7B, 2-bit | perplexity: 6.28 -> 6.26 | Mixed |
| increase calibration tokens from 4M to 8M (ablation) | GLVQ-8D · WikiText-2, Llama 1-13B, 2-bit | perplexity: 5.64 -> 5.65 | No clear effect |
| replace Babai rounding with greedy coordinate descent (ablation) | GLVQ-8D · WikiText-2, Llama 1-13B, 2-bit | perplexity: 5.64 -> 6.08 | Hurt |
Surprising findings
- Ablations show a fixed shared lattice and GCD index assignment each degrade perplexity; the GLVQ-8D-u variant recovers near-QTIP throughput.
Limitations
- Assumes a fixed column partition of groups, which may under-use layer-wise sensitivity structure.
- Only weights are compressed; activations are not quantized.
- Throughput is lower than QTIP for the mixed-precision GLVQ variants (82 vs 105 TOK/s).
Experimental setup
Datasets: RedPajama
Benchmarks: WikiText-2 C4 ARC-Challenge ARC-Easy PIQA Winogrande
Metrics: perplexity zero-shot accuracy tokens per second memory bandwidth
Baselines: OmniQuant AWQ QuIP# AQLM QTIP GPTVQ PV-Tuning BiLLM OneBit PB-LLM
Builds on: QuIP# AQLM Slim-LLM QTIP Babai rounding
Closest papers by meaning
- QuIP$\#$: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks (ICML 2024 · similarity 0.90)
- LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.90)
- Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment (NeurIPS 2025 · similarity 0.89)
- PB-LLM: Partially Binarized Large Language Models (ICLR 2024 · similarity 0.89)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.89)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.