Research map / Architectures and efficiency / Quantization

RSQ: Learning from Important Tokens Leads to Better Quantized LLMs

NeurIPS 2026 · Architectures and efficiency › Quantization · approach: KV cache quantization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RSQ, a layer-wise PTQ method that rotates weights (QuaRot-style Hadamard), scales token features by attention-based importance, then quantizes with GPTQ using importance-weighted Hessians, showing gains over QuaRot and GPTQ on LLaMA3, Mistral-NeMo, and Qwen2.5.

Problem

Layer-wise post-training quantization treats all tokens equally in the layer reconstruction loss, even though LLMs attend unequally to tokens; the paper asks whether prioritizing important tokens yields better quantized LLMs, especially at low bit widths.

Approach

Reweight each token's input features by an importance score (best: attention concentration, the column sum of attention probabilities) before GPTQ, so the Hessian becomes 2XR^2X^T, combined with rotation to remove outliers and a data-expansion (shift) augmentation to avoid positional bias.

Key results

• At 3-bit on LLaMA3-8B-Instruct, RSQ reaches 65.4% average accuracy over 10 tasks vs 63.8% for QuaRot and WikiText perplexity 9.046 vs 9.517.
• RSQ improves average accuracy over QuaRot by 1.6, 0.9, and 0.4 points on LLaMA3-8B, Mistral-NeMo-12B, and Qwen2.5-7B respectively.
• The gap widens at lower precision: at 2-bit on LLaMA3, RSQ gets WikiText PPL 16.26 vs 22.71 for QuaRot and 40.6% vs 35.3% average accuracy.

Surprising findings

• Quantizing with only the first 1024 of 4096 tokens yields better results than using all tokens.
• Scaling token importance is far more effective when applied to rotated weights than unrotated ones (optimal r_min 0.005 vs 0.1).

Limitations

• Gains on some tasks are small or within variance, e.g. Qwen2.5 RSQ is equal to QuaRot on average within noise.
• Long-context gains are inconsistent; matching calibration length to downstream length did not clearly help.
• Hyperparameters such as r_min and expansion factor M were tuned mainly on WikiText-2 perplexity.

Experimental setup

Datasets: WikiText-2 RedPajama C4 PTB LAMBADA WinoGrande ARC HellaSwag PIQA MMLU GSM8k TruthfulQA

Benchmarks: WikiText-2 LAMBADA WinoGrande ARC HellaSwag PIQA MMLU GSM8k TruthfulQA LITM LongEval L-Eval

Metrics: perplexity accuracy F1 ChrF

Baselines: GPTQ QuaRot

Builds on: GPTQ QuaRot OBC SliceGPT

Closest papers by meaning

• GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (NeurIPS 2026 · similarity 0.90)
• Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.89)
• CBQ: Cross-Block Quantization for Large Language Models (ICLR 2025 · similarity 0.89)
• SpinQuant: LLM Quantization with Learned Rotations (ICLR 2025 · similarity 0.88)
• ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals (ICML 2025 · similarity 0.88)
• OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models (AAAI 2024 · similarity 0.88)