Research map / Architectures and efficiency / RNNs and normalization
Error Feedback Can Accurately Compress Preconditioners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EFCP, which applies error feedback to gradients before they enter full-matrix preconditioner sliding windows (M-FAC, GGT), compressing the window via sparsity or low rank with GPU kernels for sparse ring buffers.
Problem
Full-matrix preconditioners such as GGT and M-FAC need to store a sliding window of m past gradients (memory Θ(md)), which is prohibitive even for small models.
Approach
Compress gradients (Top-k sparsification or low-rank power iteration) before inserting them into the preconditioner's gradient window, and feed the compression error back into subsequent gradients via error feedback, so the window stores only compressed representations while preserving accuracy.
Key results
- Sparse M-FAC at 1% density reaches 70.01% top-1 on ImageNet ResNet-18, matching Dense M-FAC (70.03%) and beating SGD (69.12%), using 14.3 GB vs 58.5 GB for Dense M-FAC.
- On BERT-Base GLUE-MNLI, S-MFAC reaches 84.86 accuracy vs 84.6 for Adam, while Dense M-FAC runs out of memory.
- Theoretical memory savings are about 45.5x (float values) to 58.5x (bfloat16) for Sparse M-FAC vs Dense M-FAC and about 33.3x for Sparse GGT vs Dense GGT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compress preconditioner gradients with 1% density and error feedback | M-FAC on ImageNet-1K ResNet-18 · ImageNet-1K validation | Top-1 accuracy: 70.03% -> 70.01% | No clear effect |
| replace Dense M-FAC with sparse M-FAC using 1% gradient density | M-FAC on ImageNet-1K ResNet-18 · ImageNet-1K | memory: 58.5 GB -> 14.3 GB | Helped |
| compress preconditioner gradients with sparsity and error feedback | M-FAC on GLUE-MNLI BERT-Tiny · GLUE-MNLI, BERT-Tiny | evaluation accuracy: 66.86 -> 71.06 | Helped |
| replace Dense M-FAC with sparse M-FAC | M-FAC on GLUE-MNLI BERT-Mini · GLUE-MNLI, BERT-Mini | evaluation accuracy: 76.39 -> 76.40 | No clear effect |
| use sparse M-FAC instead of Adam | BERT-Base · GLUE-MNLI | evaluation accuracy: 84.6 -> 84.86 | Helped |
| use sparse M-FAC instead of Dense M-FAC | BERT-Base · GLUE-MNLI | whether training runs: OOM -> 84.86 | Helped |
| compress gradients with sparsity and error feedback | M-FAC on CIFAR-10 ViT-T · CIFAR-10, ViT-T | test accuracy: 98.0 -> 98.0 | No clear effect |
| reduce gradient density to k=0.17% (ablation) | S-MFAC on ImageNet-1K ResNet-18 · ImageNet-1K validation | Top-1 accuracy | Mixed |
| use sparse M-FAC instead of SGD | ResNet-20 · CIFAR-10 | maximum Hessian eigenvalue: 9.12 / 24.51 -> 0.06 / 0.11 | Helped |
| vary gradient density from k=0.1% to k=10% (ablation) | S-MFAC · ResNet-20, CIFAR-10, epoch 100 | maximum Hessian eigenvalue (train / test): 0.1 / 0.23; 0.07 / 0.1; 0.06 / 0.11; 0.05 / 0.11; 0.05 / 0.12 | Mixed |
| compress gradients with the proposed sparse data structures and CUDA kernels rather than a naive PyTorch implementation | Sparse M-FAC · implementation efficiency | speed: 1.5-2x faster | Helped |
Surprising findings
- Sparse M-FAC at 0.17% gradient density still achieves accuracy comparable to SGD on ImageNet ResNet-18.
- Sparse M-FAC yields flatter minima (lower max Hessian eigenvalue) than SGD on ResNet-20/CIFAR-10.
Limitations
- Current implementation has constant runtime overhead vs highly optimized Adam/SGD, notably on BERT models.
- Theoretical convergence analysis is only partial; full-matrix case relies on additional assumptions about Hessian approximation stability.
- CUDA kernels were primarily optimized for residual networks and are not adapted to BERT.
Experimental setup
Datasets: ImageNet-1K CIFAR-10 GLUE MNLI
Benchmarks: ImageNet-1K CIFAR-10 GLUE MNLI
Metrics: top-1 validation accuracy test accuracy training loss peak GPU memory training time max eigenvalue of Hessian
Baselines: SGD with momentum AdamW Dense M-FAC Shampoo K-FAC SENG Adam Dense GGT
Builds on: M-FAC GGT Adagrad error feedback ASDL
Closest papers by meaning
- Extreme Tensoring for Low-Memory Preconditioning (ICLR 2020 · similarity 0.88)
- The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton (ICLR 2026 · similarity 0.87)
- Learning Sparse Approximate Inverse Preconditioners for Conjugate Gradient Solvers on GPUs (NeurIPS 2025 · similarity 0.87)
- Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.86)
- Universal Sequence Preconditioning (NeurIPS 2025 · similarity 0.86)
- On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.