Research map / Architectures and efficiency / Quantization
Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rotated Runtime Smooth (RRS), a training-free activation smoother that combines runtime channel-wise smoothing with Hadamard rotation to enable accurate INT4 weight-activation inference.
Problem
Activation outliers (channel-wise and spike outliers) stretch quantization ranges and hinder accurate INT4 weight-activation (A4W4) quantization of LLMs, while existing smoothing (SmoothQuant) and rotation (QuaRot) methods fail or are insufficient at 4 bits.
Approach
Runtime Smooth divides activations by group-wise channel maxima computed at inference (after reordering channels), avoiding migration of outliers into weights; applying it after a Hadamard rotation spreads spike outliers across channels so the smoothing scales become consistent, removing the victim effect.
Key results
- On LLaMA3-70B under A4W4KV16, RRS reduces WikiText-2 perplexity from 57.33 (QuaRot) to 6.66.
- RRS achieves about 3% average zero-shot accuracy gain over QuaRot on Common Sense QA (e.g., LLaMA-3-8B avg 58.0 vs 55.0 under 4-4-16).
- Runtime Smooth alone with group size 1 reduces LLaMA3-8B A4W16 perplexity from ~4e2 to 10.9, and RRS outperforms SpinQuant on LLaMA-2/3 models in perplexity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RRS instead of QuaRot | INT4 inference · WikiText-2, A4W4KV16, LLaMA3-70B | perplexity: 57.33 -> 6.66 | Helped |
| Apply RRS instead of QuaRot | INT4 inference · WikiText-2, A4W4KV16, LLaMA2-70B | perplexity: 3.85 -> 3.86 | Hurt |
| Apply RRS instead of QuaRot | INT4 inference · WikiText-2, A4W4KV16, LLaMA3-8B | perplexity: 8.38 -> 8.11 | Helped |
| Apply RRS instead of QuaRot | INT4 inference · Common Sense QA, 4-4-16, LLaMA-3.1-8B | Avg. accuracy (%): 57.7 -> 61.7 | Helped |
| Apply RRS instead of QuaRot | INT4 inference · Common Sense QA, 4-4-4, Mistral | Avg. accuracy (%): 62.9 -> 64.4 | Helped |
| Increase RRS runtime smoothing group size from 1 to 128 (ablation) | RRS · WikiText-2, Mistral | perplexity: 6.29 -> 6.31 | Hurt |
| Increase RRS runtime smoothing group size from 1 to 512 (ablation) | RRS · WikiText-2, Mistral | perplexity: 6.29 -> 6.33 | Hurt |
| Increase runtime smoothing group size (ablation) | Runtime Smooth · WikiText-2, Mistral, Qwen1.5-7B, and LLaMA3.1-8B | perplexity | Hurt |
| Use RRS rather than RS as group size increases (ablation) | Runtime smoothing on WikiText-2 · WikiText-2, Mistral, Qwen1.5-7B, and LLaMA3.1-8B | perplexity | Helped |
| Replace fixed rotation with trainable rotation (ablation) | SpinQuant compared with the training-free method · WikiText-2, LLaMA-2-7B, LLaMA-2-13B, LLaMA-3-8B, and LLaMA-3.1-8B | perplexity: 6.03 -> 6.37; 5.35 -> 5.55; 7.91 -> 7.99; 7.89 -> 7.93 | Hurt |
Surprising findings
- Runtime Smooth alone with group size 128 degrades sharply (e.g., LLaMA3.1-8B perplexity 10.39 at group 1 to 51.63 at group 128), showing that spike outliers make coarse grouping fail without rotation.
- The training-based SpinQuant underperforms the training-free QuaRot and RRS on WikiText-2 perplexity in the authors' re-implementation.
Limitations
- Rotated Runtime Smooth can still be vulnerable when only two outlier tokens appear in one activation (per the appendix analysis).
- Weight quantization via GPTQ fails on some models (e.g., LLaMA3-70B under RS/SmoothQuant), so A4W16 results are reported to isolate activation effects.
- Runtime Smooth without rotation is highly sensitive to group size, and the large-group setting requires rotation.
Experimental setup
Datasets: WikiText-2
Benchmarks: WikiText-2 ARC-e ARC-c BoolQ OBQA
Metrics: perplexity zero-shot accuracy kernel latency
Baselines: RTN SmoothQuant GPTQ QuaRot SpinQuant Runtime Smooth (RS)
Builds on: SmoothQuant QuaRot GPTQ QuIP
Closest papers by meaning
- ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs (NeurIPS 2026 · similarity 0.89)
- Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models (ICLR 2024 · similarity 0.89)
- ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization (AAAI 2025 · similarity 0.88)
- ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals (ICML 2025 · similarity 0.88)
- SpinQuant: LLM Quantization with Learned Rotations (ICLR 2025 · similarity 0.88)
- Training-Free Activation Sparsity in Large Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.