Research map / Architectures and efficiency / Quantization

OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models

AAAI 2024 · Architectures and efficiency › Quantization · approach: Post-training quantization of LLMs · 72 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OWQ, a sensitivity-aware mixed-precision weight quantization that keeps a small set of outlier-sensitive weight columns in fp16 and quantizes the rest with OPTQ, plus Weak Column Tuning (WCT) for parameter-efficient fine-tuning of the quantized model.

Problem

Post-training weight quantization of LLMs to ~3 bits loses substantial accuracy, and quantized-base PEFT (e.g., QLoRA) is hampered by low-quality quantized dense weights; activation outliers make certain weight columns highly sensitive to quantization.

Approach

Select weight columns by a Hessian-weighted perturbation sensitivity (lambda_j ||ΔW_:,j||^2), keep those weak columns in fp16 while quantizing the rest with OPTQ plus grid-searched truncated ranges, and fine-tune only the retained fp16 weak columns (WCT) on top of the frozen low-bit matrix.

Key results

• 3.1-bit OWQ achieves OPT-6.7B WikiText-2 perplexity 11.14 vs 12.88 for 3-bit OPTQ and 11.10-level comparable to 4-bit OPTQ (10.86 full-precision reference 10.86 vs OPTQ 4-bit 11.14)
• On LLaMA-7B WikiText-2, 3.01-bit OWQ gives 6.66 perplexity vs 8.13 for 3-bit OPTQ and 6.10 for 4-bit OPTQ
• WCT with 8 weak columns on LLaMA-7B (4-bit base) matches full-precision LoRA in GPT-4 Vicuna evaluation and beats QLoRA, using only 6.8% of QLoRA's learnable parameters; OWQ+WCT inference memory is 24.4% of full-precision LoRA's

Measured effects reported in the paper

Surprising findings

• Applying truncation (range clipping) to standard OPTQ sharply degrades OPT-6.7B WikiText-2 perplexity from 12.88 to 48.26, but OWQ avoids this because weak columns stay in fp16
• Act-order and true-sequential options that boost OPTQ provide no benefit when applied on top of OWQ

Limitations

• Uniform per-layer allocation of weak columns is suboptimal; layer-wise sensitivity varies and tuning per-layer budgets is expensive
• Fine-tuning evaluation relies on GPT-4 judging over only 80 Vicuna questions, and the QLoRA comparison uses an official checkpoint
• Weak-column storage requires a custom kernel for acceleration; overhead is small but non-zero (about 3% kernel latency on LLaMA-7B)

Experimental setup

Datasets: C4 WikiText-2 Penn Treebank OpenAssistant Vicuna Benchmark LAMBADA Winogrande ARC-challenge HellaSwag MMLU

Benchmarks: WikiText-2 Penn Treebank C4 ARC-challenge HellaSwag MMLU LAMBADA Winogrande Vicuna Benchmark

Metrics: perplexity accuracy GPT-4 preference win count latency overhead memory overhead

Baselines: RTN OPTQ QLoRA LoRA AWQ-style outlier-magnitude methods

Builds on: OPTQ Optimal Brain Compression QLoRA LoRA

Closest papers by meaning

• OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting (ICLR 2025 · similarity 0.92)
• QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.92)
• Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models (ICLR 2024 · similarity 0.91)
• LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.91)
• ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals (ICML 2025 · similarity 0.90)
• Treasures in Discarded Weights for LLM Quantization (AAAI 2025 · similarity 0.90)