AI Paper Atlas

Research map / Optimization / Implicit bias and generalization

Iterate Averaging in the Quest for Best Test Error

Diego Granziol, Nicholas P. Baskerville, Xingchen Wan, Samuel Albanie, Stephen Roberts

JMLR 2024 · Optimization › Implicit bias and generalization · approach: SGD noise and flat minima · 11 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add iterate averaging (GadamX vs AdamW) with partially decoupled weight decayAdamW · CIFAR-100, ResNeXt-29test accuracy: 80.16 -> 83.27Improved
add iterate averaging and strong weight decay to Adam (Gadam vs AdamW)AdamW · ImageNet, ResNet-50top-1 accuracy: 74.04 -> 76.79Improved
add iterate averaging to SGD (SWA vs SGD)SGD · ImageNet 32x32, WRN-28-10top-1 accuracy: 61.33 -> 62.32Improved
use GadamX instead of SWASWA · CIFAR-100, VGG-16test accuracy: 74.57 -> 76.85Improved
use GadamX instead of SWASWA · CIFAR-100, PRN-110test accuracy: 77.92 -> 77.90No clear effect
use plain Gadam instead of tuned SGDSGD · ImageNet 32x32, WRN-28-10top-1 accuracy: 61.33 -> 60.50Hurt
tune the averaging start T_avg (ablation)GadamX · CIFAR-100, PRN-110test accuracy: 77.90 -> 79.41Improved
tune the averaging start T_avg (ablation)GadamX · CIFAR-100, VGG-16test accuracy: 77.90 -> 77.22Improved
reduce numerical stability constant to 1e-4 in GadamX (partial adaptivity) (ablation)GadamX · ImageNet, ResNet-50top-1 accuracy: 77.19 -> 77.31Improved
raise base learning rate to 0.003 in SGD-like Gadam (ablation)Gadam · ImageNet, ResNet-50top-1 accuracy: 75.52 -> 76.53Improved
average once per iteration instead of less often (ablation)Gadam · PTB LSTM validation perplexityperplexityNo clear effect
use automatic averaging trigger and early stopping (Auto) with Adam base vs SGD base (ablation)Gadam-Auto vs SWA-Auto · CIFAR-100, VGG-16test accuracy: 73.93 -> 75.39Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy