Research map / Architectures and efficiency / Quantization
OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OSTQuant, a PTQ method that learns orthogonal and scaling equivalent transformations to fit weight/activation distributions across the quantization space, guided by a new QSUR metric and a top-k KL loss (KL-Top).
Problem
Post-training quantization of LLMs suffers from uneven, heavy-tailed weight and activation distributions that expand the quantization range; prior linear-transformation methods are heuristic and do not optimize the distribution across the whole quantization space, and small calibration sets make cro
Approach
Introduce QSUR (ratio of data hypervolume to quantization hypercube volume) as a quantizability metric, and learn per-layer equivalent transformation pairs combining Stiefel-manifold orthogonal matrices (RiemannAdam) with diagonal scaling, initialized by WOMI; train with a KL divergence restricted to the top-k teacher logits to avoid long-tail noise.
Key results
- In W4-only (4-16-16) setting, OSTQuant retains at least 99.5% of FP zero-shot accuracy; on LLaMA-3-8B it drops only 0.29 points vs over 1.55 for other methods.
- In W4A4KV4 on LLaMA-3-8B, OSTQuant reduces the performance gap by 32% relative to SOTA methods.
- Prefill speedup of about 2x and memory saving above 3.5x on a 3090 GPU for 4-bit inference; 7B/13B optimized in about 20 minutes, up to 5.3x faster than OmniQuant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OSTQuant instead of SpinQuant | LLaMA-3-8B W4A4KV4 · LLaMA-3 8B W4A4KV4 | zero-shot average accuracy: 64.10 -> 65.37 | Helped |
| Add KL-Top loss to OSTQuant (ablation) | LLaMA-3-8B W4A4KV4 · LLaMA-3 8B W4A4KV4 | zero-shot average accuracy: 65.13 -> 65.37 | Mixed |
| Add KL-Top loss to OSTQuant (ablation) | LLaMA-2-7B W4A4KV4 · LLaMA-2 7B W4A4KV4 | zero-shot average accuracy: 62.45 -> 63.11 | Helped |
| Add KL-Top loss to SpinQuant (ablation) | LLaMA-3-8B W4A4KV4 · LLaMA-3 8B W4A4KV4 | zero-shot average accuracy: 64.10 -> 64.07 | Hurt |
| Set KL-Top k to 1000 (ablation) | LLaMA-2-7B W4A4KV4 · LLaMA-2 7B W4A4KV4 | zero-shot average accuracy: 62.4 -> 63.18 | Helped |
| Set KL-Top k to 1000 (ablation) | LLaMA-2-7B W3-only · LLaMA-2 7B W3-only | zero-shot average accuracy: 61.87 (k=5) -> 62.30 (k=1000); 61.25 (k=5000) | Mixed |
| Use WOMI instead of Hadamard initialization (ablation) | LLaMA-2-7B W4A16KV16 · LLaMA-2 7B W4A16KV16 | zero-shot average accuracy: 63.32 -> 63.45 | Helped |
| Use WOMI instead of Hadamard initialization (ablation) | LLaMA-2-7B W4A4KV4 · LLaMA-2 7B W4A4KV4 | zero-shot average accuracy: 61.47 -> 61.52 | Helped |
| Use RiemannAdam instead of Cayley SGD (ablation) | LLaMA-2-7B W4A4KV4 · LLaMA-2 7B W4A4KV4 | zero-shot average accuracy: 63.11 -> 63.18 | Helped |
| Use RiemannAdam instead of Riemann SGD (ablation) | LLaMA-2-13B W4A4KV4 · LLaMA-2 13B W4A4KV4 | zero-shot average accuracy: 65.19 -> 65.41 | Helped |
| Add R_dn after R_res and S_res (ablation) | LLaMA-2-7B W4A4KV4 · LLaMA-2 7B W4A4KV4 | zero-shot average accuracy: 53.74 -> 61.75 | Helped |
| Add S_u|d after R_dn (ablation) | LLaMA-2-7B W4A4KV4 · LLaMA-2 7B W4A4KV4 | zero-shot average accuracy: 61.75 -> 61.79 | Helped |
Surprising findings
- Applying the KL-Top loss to SpinQuant alone gives no meaningful gain and sometimes degrades results, while combined with OST transformations it helps.
- Standard cross-entropy calibration on small data lowers WikiText perplexity yet reduces zero-shot accuracy, indicating overfitting.
Limitations
- Full quantization of activations within blocks (e.g., ROPE, SiLU) is left as future work.
- QSUR derivations rely on Gaussian assumptions and neglect the mean vector.
- Only uniform quantization and per-channel/per-token settings are studied.
Experimental setup
Datasets: WikiText2
Benchmarks: WikiText2 BoolQ HellaSwag LAMBADA OpenBookQA PIQA SIQA WinoGrande ARC-Easy ARC-Challenge
Metrics: zero-shot accuracy average perplexity speedup memory saving
Baselines: RTN SmoothQuant GPTQ OmniQuant AWQ QuaRot SpinQuant
Builds on: QuaRot SpinQuant SmoothQuant OmniQuant RiemannAdam
Closest papers by meaning
- AffineQuant: Affine Transformation Quantization for Large Language Models (ICLR 2024 · similarity 0.92)
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.92)
- OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models (AAAI 2024 · similarity 0.92)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.92)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.92)
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.