Research map / Architectures and efficiency / Quantization
QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QLLM, a post-training quantization method for LLMs that disassembles outlier activation channels into sub-channels, re-merges similar channels to keep channel count, and tunes low-rank adapters that are merged into quantized weights afterward.
Problem
Activation outlier channels in LLMs make low-bit (e.g., W4A4) post-training quantization severely degrade accuracy; existing smoothing methods either offer limited relief or suffer unstable gradients, and block-wise full-weight tuning is memory-expensive.
Approach
Adaptive channel reassembly: split outlier input channels into T equal-output sub-channels (with duplicated weights), then merge T-1 most similar channel pairs via bipartite soft matching to keep the channel count; the threshold is chosen per layer by grid search minimizing attention/FFN output reconstruction error, followed by low-rank error correction (EEC) merged back into weights.
Key results
- On 4-bit LLaMA-2-70B, QLLM reaches 58.62% average zero-shot accuracy versus 50.73% for OS+ and 48.64% for SmoothQuant (7.89% gain over SmoothQuant), quantized in under 10 hours on a single A100-80G.
- On W4A4 LLaMA-1-7B, QLLM achieves 51.84% average accuracy vs 50.47% for OmniQuant and 36.28% for SmoothQuant, and surpasses LLM-QAT+SQ by about 8.6 points.
- QLLM W4A4 LLaMA-1-65B perplexity on WikiText2 is 6.87 vs 9.17 for OmniQuant; 4-bit LLaMA-1-13B throughput on RTX 3090 is 1.96x faster than FP16 with QUIK W4A4 kernels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QLLM instead of OmniQuant for W4A4 quantization | LLaMA-1-7B · five zero-shot tasks | accuracy Avg. (%): 50.47 -> 51.84 | Helped |
| Use QLLM instead of OmniQuant for W4A4 quantization | LLaMA-1-65B · five zero-shot tasks | accuracy Avg. (%): 56.41 -> 59.83 | Helped |
| Use QLLM instead of SmoothQuant for W4A4 quantization | LLaMA-2-70B · five zero-shot tasks | accuracy Avg. (%): 48.64 -> 58.62 | Helped |
| Add channel disassembly to the quantized model (ablation) | LLaMA-1-13B W4A4 · WikiText2 | perplexity: 1702.34 -> 19.34 | Helped |
| Increase channel expansion ratio from γ=0.01 to γ=0.07 (ablation) | LLaMA-1-13B W4A4 with channel disassembly · WikiText2 | perplexity: 8.31 -> 7.81 | Helped |
| Add channel assembly to channel disassembly at γ=0.07 (ablation) | LLaMA-1-13B W4A4 · WikiText2 | perplexity: 7.81 -> 9.39 | Hurt |
| Use adaptive reassembly instead of fixed-ratio disassembly and assembly (ablation) | LLaMA-1-13B W4A4 · WikiText2 | perplexity: 8.68 -> 8.41 | Helped |
| Add efficient error correction to channel disassembly at γ=0.07 (ablation) | LLaMA-1-13B W4A4 · WikiText2 | perplexity: 11.13 -> 7.81 | Helped |
| Add channel reassembly to EEC (ablation) | LLaMA-1-7B W4A4 · WikiText2 | perplexity: 16.64 -> 9.65 | Helped |
| Add adaptive channel reassembly to TQW (ablation) | LLaMA-1-7B W4A4 · WikiText2 | perplexity: 13.13 -> 8.90 | Helped |
| Add channel reassembly (ablation) | LLaMA-1-7B · calibration set block reconstruction | reconstruction error: 4.71 -> 2.74 | Helped |
| Use EEC instead of tuning all quantized weights directly (ablation) | LLaMA-1-7B W4A4, 1 reconstruction block · WikiText2 | perplexity: 10.10 -> 11.21 | Hurt |
Surprising findings
- Channel reassembly without efficient error correction still yields large gains; EEC alone (without reassembly) at W4A4 LLaMA-1-7B gives perplexity around 28 average, far worse than QLLM.
- Merging low-rank weights into the frozen weights via quant(W+AB) slightly increases perplexity (e.g., LLaMA-1-7B avg 12.40 to 12.83) but avoids inference overhead.
Limitations
- Channel disassembly and assembly add runtime operations for layers preceded by non-linear layers (e.g., LayerNorm), incurring inference overhead and non-GPU-friendly channel counts.
- Evaluation focuses on LLaMA-1/2 families with W4A4, W6A6, W4A8 settings; W6A6 gains over OmniQuant are modest.
- Relies on a calibration set from WikiText2 and a grid-searched per-layer threshold, adding offline calibration cost.
Experimental setup
Datasets: WikiText2 C4 PTB
Benchmarks: PIQA ARC-e ARC-c HellaSwag Winogrande WikiText2 C4 PTB Vicuna benchmark
Metrics: zero-shot accuracy perplexity throughput (tokens/s) win/tie/lose counts
Baselines: OmniQuant SmoothQuant Outlier Suppression+ LLM-QAT QUIK
Builds on: OmniQuant SmoothQuant LoRA bipartite soft matching
Closest papers by meaning
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.92)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.92)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.92)
- Evaluating Quantized Large Language Models (ICML 2024 · similarity 0.92)
- CBQ: Cross-Block Quantization for Large Language Models (ICLR 2025 · similarity 0.92)
- OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.