AI Paper Atlas

Research map / Optimization / Implicit bias and generalization

Entropy-SGD: Biasing Gradient Descent Into Wide Valleys

Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, Riccardo Zecchina

ICLR 2017 · Optimization › Implicit bias and generalization · approach: SGD noise and flat minima · 104 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Optimize local entropy with Entropy-SGD rather than the standard objectivemnistfc on MNIST · MNISTtest error (%): 1.39 ± 0.03 -> 1.37 ± 0.03No clear effect
Optimize local entropy with Entropy-SGD rather than the standard objectiveLeNet · MNISTtest error (%): 0.51 ± 0.01 -> 0.5 ± 0.01No clear effect
Optimize local entropy with Entropy-SGD rather than the standard objectiveAll-CNN-BN · CIFAR-10test error (%): 7.71 ± 0.19 -> 7.81 ± 0.09No clear effect
Optimize local entropy with Entropy-SGD rather than the standard objectivePTB-LSTM · PTB testperplexity: 78.6 ± 0.26 -> 77.656 ± 0.171Improved
Optimize local entropy with Entropy-Adam rather than Adamchar-LSTM · War and Peace testperplexity: 1.226 ± 0.01 -> 1.217 ± 0.005Improved
Use Entropy-SGD instead of SGDPTB-LSTM · PTBepochs: 55 -> 25Improved
Use Entropy-Adam instead of Adamchar-LSTM · War and Peaceepochs: 40 -> 25Improved
Replace the local-entropy objective with vanilla SGLDLeNet · MNIST testtest error (%): 0.50 ± 0.01 -> 0.63 ± 0.1Hurt
Replace the local-entropy objective with vanilla SGLDAll-CNN-BN · CIFAR-10 testtest error (%): 7.81 ± 0.09 -> 9.89 ± 0.11Hurt
Replace the local-entropy objective with vanilla SGLDPTB-LSTM · PTB testperplexity: 77.656 ± 0.171 -> 94.03Hurt
Replace the local-entropy objective with vanilla SGLDchar-LSTM · War and Peace testperplexity: 1.213 ± 0.007 -> 1.3375Hurt
Anneal the SGLD learning rate (including matching it to the outer learning rate) (ablation)Entropy-SGD · Deep-network experimentslocal-entropy gradient estimate qualityHurt

Later papers that compared with this method

1 later accepted paper reports Entropy-SGD as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy