Research map / Architectures and efficiency / Quantization

Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models

AAAI 2024 · Architectures and efficiency › Quantization · approach: Post-training quantization of LLMs · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Norm-Tweaking, a plug-in PTQ step that tunes only LayerNorm/RMSNorm parameters with a channel-wise mean/variance distribution loss and self-generated calibration data, to recover accuracy in 2-bit and W4A8 quantization of LLMs.

Problem

Low-bit (2-bit weight-only and W4A8) post-training quantization of LLMs causes severe accuracy degradation, with quantized activation distributions drifting from float ones layer by layer.

Approach

Adjust only the LayerNorm (or RMSNorm) parameters of each quantized transformer block, after GPTQ quantization, to match per-channel mean and variance of the float model's activations, using a small learning rate, one iteration over LLM-generated calibration text, and a layer-wise increasing learning rate schedule.

Key results

• On LAMBADA at 2-bit, Norm-Tweaking lifts LLaMA-65b from 57.15 (GPTQ) to 67.48, and OPT-66b reaches 73.49 vs 73.30 FP16, with GLM-130b at 69.43 vs 69.42 FP16.
• Quantization runtime overhead is modest: BLOOM-7B GPTQ 19.6 min vs GPTQ+NT 22.8 min on a single A100.
• Applied to RTN W4A16 on BLOOM-7B, LAMBADA accuracy rises from 48.36 to 51.56.

Measured effects reported in the paper

Surprising findings

• Random Gaussian calibration data gives extremely poor quantization quality, while LLM-generated data works well.
• Increasing tweaking iterations sharply degrades accuracy (BLOOM-7B LAMBADA drops from 57.5 at 1 iteration to 11.3 at 50), so only one iteration is used.

Limitations

• Some benchmarks (e.g., CB, RTE, QNLI in the appendix) do not improve and the paper does not explain why.
• Tuning is sensitive: learning rate and iteration count must be grid-searched, and excessive tuning collapses performance.
• Evaluation is mostly on LAMBADA and few-shot harness tasks; the main method is built on GPTQ with symmetric per-channel quantization.

Experimental setup

Datasets: LLM-generated calibration data WikiText-2 PTB C4

Benchmarks: LAMBADA HellaSwag PIQA WinoGrande OpenBookQA RTE MRPC QNLI BoolQ CB COPA WiC

Metrics: accuracy perplexity

Baselines: GPTQ SmoothQuant RTN OmniQuant AWQ ZeroQuant-V2

Builds on: GPTQ SmoothQuant LLM-QAT RTN

Closest papers by meaning

• Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.93)
• QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.92)
• ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization (AAAI 2025 · similarity 0.92)
• SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models (ICML 2025 · similarity 0.92)
• CBQ: Cross-Block Quantization for Large Language Models (ICLR 2025 · similarity 0.92)
• Compressing Large Language Models by Joint Sparsification and Quantization (ICML 2024 · similarity 0.92)