Research map / Architectures and efficiency / RNNs and normalization
DASH: Faster Shampoo via Batched Block Preconditioning and Efficient Inverse-Root Solvers
Ionut-Vlad Modoranu, Philip Zmushko, Erik Schultheis, Mher Safaryan, Dan Alistarh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| batch block preconditioners (DASH) instead of sequential Distributed Shampoo, CN FP32 | Distributed Shampoo · Llama-953M C4, B=1024 | optimizer step ms: 666 -> 136 | Improved |
| DASH with CN FP16 versus DIST CN FP32 at B=1024 | Distributed Shampoo · Llama-953M C4, B=1024 | optimizer step ms: 666 -> 119 | Improved |
| DASH on EVD f=1, B=2048 | Distributed Shampoo EVD · Llama-953M C4 | optimizer step ms: 2200 -> 1747 | Improved |
| DASH versus DIST on validation perplexity (EVD, B=2048, f=1) | Distributed Shampoo EVD · Llama-953M C4 | validation perplexity: 11.72 -> 11.73 | No clear effect |
| use NDB with Power-Iteration scaling instead of Frobenius scaling (ablation) | DASH NDB · Llama-953M C4, B=1024 | validation perplexity: 11.76 -> 11.68 | Improved |
| use NDB instead of CN-FP32 as inverse root | DASH · Llama-953M C4, B=1024 | validation perplexity: 11.87 -> 11.68 | Improved |
| use FP16 instead of FP32 iterations for CN (ablation) | DASH CN · Llama-953M C4, B=1024 | optimizer step ms: 136 -> 119 | Improved |
| use BF16 instead of FP16 for CN iterations (ablation) | DASH CN · Llama-953M C4 | convergence | Hurt |
| use FP16 or BF16 for NDB iterations (ablation) | DASH NDB · Llama-953M C4 | convergence | Hurt |
| increase block size from 1024 to 2048 (ablation) | DASH · Llama-953M C4 | optimizer step ms: NDB PIM 177 -> 267 | Hurt |
| replace SHMP dampening with RELU heuristic (ablation) | DASH EVD · Llama-373M | validation perplexity: 15.893 -> 15.876 | No clear effect |
| replace Distributed Shampoo with DASH on NDB, ViT-S ImageWoof | Distributed Shampoo NDB · ImageWoof ViT-S | opt step ms: 145 -> 15 | Improved |
Closest papers by meaning
- A New Perspective on Shampoo's Preconditioner (ICLR 2025 · similarity 0.91)
- Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner (NeurIPS 2025 · similarity 0.89)
- 4-bit Shampoo for Memory-Efficient Network Training (NeurIPS 2024 · similarity 0.85)
- Structured Preconditioners in Adaptive Optimization: A Unified Analysis (ICML 2025 · similarity 0.84)
- A Stable Whitening Optimizer for Efficient Neural Network Training (NeurIPS 2025 · similarity 0.82)
- Learning Sparse Approximate Inverse Preconditioners for Conjugate Gradient Solvers on GPUs (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
