Research map / Architectures and efficiency / RNNs and normalization
Multiplication-Free Transformer Training via Piecewise Affine Operations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace matrix multiplications with PAM | DeiT-Tiny · CIFAR10 test | Top-1 accuracy: 94.8% -> 95.0[+0.2]% | No clear effect |
| Replace matrix multiplications with PAM | DeiT-Tiny · ImageNet-1k validation | Top-1 accuracy: 72.2% -> 72.2[+0.0]% | No clear effect |
| Replace forward- and backward-pass matrix multiplications with PAM | Transformer-Small · IWSLT14 DE-EN test | BLEU: 34.37 -> 34.22 | Hurt |
| Use approximate rather than exact derivatives for PAM matrix multiplications (ablation) | Transformer-Small · IWSLT14 DE-EN | BLEU: 33.4[-1.0]±0.1 -> 34.2[-0.2]±0.2 | Helped |
| Use approximate rather than exact derivatives for attention softmax (ablation) | Transformer-Small · IWSLT14 DE-EN | BLEU: 3.0[-31.4]±0.9 -> 34.3[-0.1]±0.7 | Helped |
| Use approximate rather than exact derivatives for layer normalization (ablation) | Transformer-Small · IWSLT14 DE-EN | BLEU: 33.7[-0.7]±0.2 -> 34.4[-0.0]±0.1 | Helped |
| Use approximate rather than exact derivatives for the loss (ablation) | Transformer-Small · IWSLT14 DE-EN | BLEU: 33.7[-0.7]±0.2 -> 33.0[-1.4]±0.9 | Hurt |
| Replace optimizer multiplications, divisions, and square roots with piecewise affine operations (ablation) | Transformer-Small · IWSLT14 DE-EN | BLEU: 34.4 -> 34.2 | Hurt |
| Replace all network operations, loss, and optimizer arithmetic with piecewise affine operations | Transformer-Small · IWSLT14 DE-EN | BLEU: 34.4±0.1 -> 33.5[-0.9]±0.3 | Hurt |
| Reduce PAM mantissa width from 7 bits to 4 bits (ablation) | VGG-13 · CIFAR-10 test | Top-1 accuracy: 92.9±0.2% -> 92.9±0.5% | No clear effect |
| Reduce PAM mantissa width from 7 bits to 4 bits (ablation) | Transformer-Small · IWSLT14 DE-EN | BLEU: 34.4±0.2 -> 34.2±0.1 | No clear effect |
| Reduce PAM mantissa width to 3 bits (ablation) | Transformer-Small · IWSLT14 DE-EN | BLEU: 34.4±0.2 -> 29.4±0.5 | Hurt |
Closest papers by meaning
- FrameQuant: Flexible Low-Bit Quantization for Transformers (ICML 2024 · similarity 0.85)
- Two Heads are Better than One: Simulating Large Transformers with Small Ones (NeurIPS 2025 · similarity 0.85)
- CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs (NeurIPS 2026 · similarity 0.85)
- Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure (NeurIPS 2024 · similarity 0.85)
- Simplifying Transformer Blocks (ICLR 2024 · similarity 0.84)
- Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.