Research map / Optimization / Optimizers
Extreme Tensoring for Low-Memory Preconditioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ET1 instead of AdaGrad to compress the adaptive preconditioner | Transformer language modeling · Google Billion Words validation | final perplexity: 41.18 -> 39.84 | Helped |
| Increase tensoring from ET1 to ET2 (ablation) | Extreme tensoring · Google Billion Words validation | final perplexity: 39.84 -> 42.10 | Hurt |
| Increase tensoring from ET1 to ET3 (ablation) | Extreme tensoring · Google Billion Words validation | final perplexity: 39.84 -> 45.19 | Hurt |
| Use compressed ET1 preconditioning instead of SGD | Transformer language modeling · Google Billion Words validation | final perplexity: 49.80 -> 39.84 | Helped |
| Use ET1 instead of Adam | ResNet-18 · CIFAR-10 test | final test error: 8.40 -> 7.22 | Helped |
| Increase tensoring from ET1 to ET2 (ablation) | Extreme tensoring · CIFAR-10 test | final test error: 7.22 -> 8.49 | Hurt |
| Increase tensoring from ET1 to ET3 (ablation) | Extreme tensoring · CIFAR-10 test | final test error: 7.22 -> 8.52 | Hurt |
| Use ET1 instead of SGD | ResNet-18 · CIFAR-10 test | final test error: 9.27 -> 7.22 | Helped |
| Use a larger 12-layer model with ET1 rather than the smaller model's fully stored AdaGrad accumulators | Transformer language modeling · Google Billion Words validation (same running time) | final perplexity: 39.25 (larger ET1) vs. 41.18 (smaller AdaGrad) | Mixed |
| Decay the second-moment estimator (ablation) | Extreme tensoring in language modeling · Google Billion Words | performance | No clear effect |
| Remove momentum (ablation) | Extreme tensoring · Large-scale language modeling | performance | No clear effect |
Closest papers by meaning
- Error Feedback Can Accurately Compress Preconditioners (ICML 2024 · similarity 0.88)
- On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning (ICML 2025 · similarity 0.88)
- Universal Sequence Preconditioning (NeurIPS 2025 · similarity 0.86)
- Structured Preconditioners in Adaptive Optimization: A Unified Analysis (ICML 2025 · similarity 0.86)
- FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training (ICML 2025 · similarity 0.86)
- Advancing the Lower Bounds: an Accelerated, Stochastic, Second-order Method with Optimal Adaptation to Inexactness (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.