Research map / Architectures and efficiency / Attention and transformers
Softmax Output Approximation for Activation Memory-Efficient Training of Attention-based Networks
Changhyeon Lee, Seulki Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| store only m=1 highest and lowest softmax outputs (11.9% memory) (ablation) | softmax approximation · AG News XLNet | f1-micro: 0.9323 -> 0.9175 | Hurt |
| increase stored elements to m=10 (20.9% memory) (ablation) | softmax approximation · AG News XLNet | f1-micro: 0.9323 -> 0.9335 | No clear effect |
| store m=30 elements (ablation) | softmax approximation · AG News XLNet | f1-micro: 0.9323 -> 0.9371 | Improved |
| m=1 on DBpedia (ablation) | softmax approximation · DBpedia XLNet | f1-micro: 0.9889 -> 0.9743 | Hurt |
| m=5 on DBpedia (15.9% memory) (ablation) | softmax approximation · DBpedia XLNet | f1-micro: 0.9889 -> 0.9869 | No clear effect |
| m=1 on SST-2 (11.9% memory) (ablation) | softmax approximation · SST-2 ALBERT | f1-micro: 0.8326 -> 0.8268 | No clear effect |
| m=40 on SST-2 (ablation) | softmax approximation · SST-2 ALBERT | f1-micro: 0.8326 -> 0.8360 | No clear effect |
| m=1 on IMDb (11.9% memory) (ablation) | softmax approximation · IMDb XLNet | f1-micro: 0.8769 -> 0.8833 | Improved |
| m=1 (2.5% memory) on Multi30k De-En (ablation) | softmax approximation · Multi30k De-En | BLEU: 36.52 -> 35.53 | Hurt |
| m=2 (14.4% memory) on De-En (ablation) | softmax approximation · Multi30k De-En | BLEU: 36.52 -> 36.29 | No clear effect |
| m=4 or more (19.4% memory) on De-En (ablation) | softmax approximation · Multi30k De-En | BLEU: 36.52 -> 36.72 | Improved |
| m=1 on Multi30k En-De (ablation) | softmax approximation · Multi30k En-De | BLEU: 34.05 -> 32.76 | Hurt |
Closest papers by meaning
- From Attention to Activation: Unraveling the Enigmas of Large Language Models (ICLR 2025 · similarity 0.85)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.85)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.85)
- MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map (NeurIPS 2024 · similarity 0.84)
- The Closeness of In-Context Learning and Weight Shifting for Softmax Regression (NeurIPS 2024 · similarity 0.84)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
