Research map / Architectures and efficiency / Quantization

Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models

ICLR 2024 · Architectures and efficiency › Quantization · approach: Post-training quantization of LLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AdaptDim, a weight-only LLM quantization scheme that groups weights per input channel instead of per output channel, adaptively choosing the grouping dimension and adding Hessian-based updates for sub-4-bit quantization.

Problem

Low-bit (sub-4-bit) weight-only quantization of LLMs suffers from large-magnitude activation outliers that concentrate in specific input channels, degrading accuracy for small-batch inference on edge devices.

Approach

Quantization groups are placed along input channels rather than output channels, since outliers occur in specific input-channel dimensions; because weight-only quantization dequantizes online, the per-output-channel hardware constraint can be relaxed. The method adaptively searches the grouping dimension and incorporates Hessian-based weight updates for sub-4-bit.

Key results

• Abstract claims the extremely lightweight per-IC RTN variant reaches on-par performance with SoTA methods at 4-bit.
• Abstract claims AdaptDim outperforms existing works on various language modeling benchmarks, with particular effectiveness on fine-tuned LLMs.

Limitations

• The provided text is only the abstract and a template; experimental details are not available.

Experimental setup

Benchmarks: language modeling benchmarks

Metrics: perplexity

Baselines: RTN GPTQ SoTA weight-only quantization methods

Builds on: GPTQ RTN round-to-nearest quantization

Closest papers by meaning

• DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs (NeurIPS 2024 · similarity 0.93)
• OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models (AAAI 2024 · similarity 0.91)
• Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.91)
• QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.91)
• ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization (AAAI 2025 · similarity 0.90)
• ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals (ICML 2025 · similarity 0.90)