Research map / Architectures and efficiency / Quantization

The Quantization Benefits of Residual-Free Transformers

NeurIPS 2026 · Architectures and efficiency › Quantization · approach: Post-training quantization of ViTs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows residual connections drive transformer activations away from Gaussianity, making low-bit quantization lossy, and that residual-free transformers trained with orthogonal init, KL Shampoo and depth-aware attention temperature stay near-Gaussian and quantize more robustly, at a small BF16 accuracy cost.

Problem

Transformer activations are heavy-tailed and outlier-dominated, making low-bit quantization for communication and deployment lossy; prior work treats this with post-hoc outlier handling, whereas this paper asks whether the architecture itself causes it.

Approach

Residual connections preserve and amplify excess kurtosis, while dense mixing without residuals contracts it; combining residual-free blocks with orthogonal initialization, spectral/second-order optimizers, and depth-aware attention temperature yields trainable residual-free transformers with near-Gaussian activations that are robust under simple uniform quantization.

Key results

• Residual + AdamW drops from 0.449 average accuracy at BF16 to 0.369 at W8A8, while residual-free + KL Shampoo reaches 0.429 at W8A8, a +2.7 point gap over the best residual model.
• At W8A6, residual-free + KL Shampoo scores 0.414 versus 0.380 for Residual + AdamW and 0.348 for Residual + KL Shampoo.
• At BF16 the residual-free model is slightly weaker (0.435 vs 0.452 for Residual + KL Shampoo), and its activations stay near-Gaussian (excess kurtosis near 0, e.g. 0.259 at layer 16).

Surprising findings

• KL Shampoo improves residual-model robustness at W8A12 but degrades sharply at W8A6 (0.452 to 0.348), so optimizer choice alone does not remove non-Gaussianity.
• Residual-free transformers trained with AdamW become increasingly hard to train as depth grows, motivating second-order optimizers.

Limitations

• Full-precision accuracy gap relative to residual models remains.
• Only tested up to 24 layers and small models; scaling to greater depth and model size is untested.
• Evaluation uses simple uniform quantization without modern outlier-handling pipelines; theory relies on idealized independence and mean-field assumptions.

Experimental setup

Datasets: FineWeb-Edu

Benchmarks: arc_c arc_e boolq hellaswag openbookqa piqa social_iqa winogrande

Metrics: zero-shot average accuracy signal-to-quantization-noise ratio excess kurtosis

Baselines: Residual + AdamW Residual + KL Shampoo

Builds on: LLM.int8() SmoothQuant Muon SOAP Shampoo FineWeb-Edu

Closest papers by meaning

• FrameQuant: Flexible Low-Bit Quantization for Transformers (ICML 2024 · similarity 0.90)
• Outlier-aware Slicing for Post-Training Quantization in Vision Transformer (ICML 2024 · similarity 0.88)
• KroQuant: Kronecker-Structured Block Transforms for Efficient Post-Training Quantization of Diffusion Transformers (NeurIPS 2026 · similarity 0.87)
• PTQ4DiT: Post-training Quantization for Diffusion Transformers (NeurIPS 2024 · similarity 0.87)
• Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization (ICML 2024 · similarity 0.87)
• Normalized Architectures are Natively 4-Bit (NeurIPS 2026 · similarity 0.87)