AI Paper Atlas

Research map / Optimization / Optimizers

A Computationally Efficient Sparsified Online Newton Method

Fnu Devvrit, Sai Surya Duvvuri, Rohan Anil, Vineet Gupta, Cho-Jui Hsieh, Inderjit S Dhillon

NeurIPS 2023 · Optimization › Optimizers · approach: Second-order and structured preconditioning · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use diagonal sparsity instead of a first-order optimizer (ablation)SONew · MNIST AutoencoderTrain CE loss: Adam 53.591 -> diag-SONew 53.025Improved
Use tridiagonal sparsity instead of diagonal sparsity (ablation)SONew · MNIST AutoencoderTrain CE loss: 53.025 -> 51.723Improved
Use banded sparsity with band size 4 instead of diagonal sparsity (ablation)SONew · MNIST AutoencoderTrain CE loss: 53.025 -> 51.357Improved
Use tridiagonal sparsity instead of AdamSONew · ImageNet ViTiterations to target validation performance: 10%Improved
Use tridiagonal sparsity instead of AdamSONew · OGBG-molpcba GraphNetworkiterations to target validation performance: 30%Improved
Use tridiagonal sparsity instead of Adam at the same number of stepsSONew · ImageNet ViTrelative validation error: ~ 0.7%Improved
Use tridiagonal sparsity instead of Adam at the same number of stepsSONew · OGBG-molpcba GraphNetworkrelative validation average precision: ~ 3.4%Improved
Use tridiagonal sparsity instead of ShampooSONew · OGBG-molpcba GraphNetworkaverage precision: 1.3%Hurt
Use tridiagonal sparsity instead of AdaFactorSONew · 1B-parameter language modelsteps to same log perplexity: 26%Improved
Use tridiagonal sparsity instead of AdaFactor with the same number of tokensSONew · 1B-parameter language modeltrain loss: 1.7%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy