Research map / Architectures and efficiency / Quantization
TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
Guang Liang, Jie Shao, Ningyuan Tang, Xinyao Liu, Jianxin Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TWEO loss to Swin-T | Swin-T · ImageNet-1K | largest outlier (peak): 1556 -> 22 | Improved |
| add TWEO loss to Swin-T | Swin-T · ImageNet-1K | top-1: 81.2 -> 81.4 | No clear effect |
| add TWEO loss to Swin-S | Swin-S · ImageNet-1K | largest outlier (peak): 6402 -> 35 | Improved |
| add TWEO loss to GPT2-XL under per-tensor FP8 | GPT2-xl · OpenWebText FP8 | validation perplexity: 93.28 -> 12.58 | Improved |
| add TWEO loss to GPT2 124M under FP8 | GPT2 124M · OpenWebText FP8 | validation perplexity: 169.81 -> 19.26 | Improved |
| add TWEO loss to GPT2 7B under FP8 | GPT2 7B · OpenWebText FP8 | validation perplexity: 67.12 -> 12.02 | Improved |
| add TWEO loss, BF16 training | GPT2-xl · OpenWebText BF16 | validation perplexity: 13.84 -> 12.39 | Improved |
| add TWEO loss to Medium before W8A8 per-tensor static quantization | GPT-2 Medium · OpenWebText W8(T)A8(T) | perplexity: 1491.11 -> 16.50 | Improved |
| quantize residual stream with SmoothQuant on default model | GPT-2 XL default · OpenWebText W8(C)A8(T) | perplexity: 14.81 -> 1876.70 | Hurt |
| apply SmoothQuant on top of TWEO model instead of AbsMax (ablation) | GPT-2 XL TWEO · OpenWebText W8(C)A8(T), residual quantized | perplexity: 13.06 -> 12.89 | No clear effect |
| use invasive attention bias instead of TWEO (ablation) | GPT2 124M · OpenWebText FP8 | validation perplexity: 19.26 (TWEO) vs 24.93 (Attn bias) | Hurt |
| random inputs to pretrained model (ablation) | Qwen2.5-0.5B · extreme outliers | outlier magnitude | No clear effect |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SmoothQuant — SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- The Quantization Benefits of Residual-Free Transformers (NeurIPS 2026 · similarity 0.89)
- Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization (ICML 2024 · similarity 0.88)
- Understanding and Minimising Outlier Features in Transformer Training (NeurIPS 2024 · similarity 0.88)
- Outlier-aware Slicing for Post-Training Quantization in Vision Transformer (ICML 2024 · similarity 0.88)
- FrameQuant: Flexible Low-Bit Quantization for Transformers (ICML 2024 · similarity 0.87)
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
