AI Paper Atlas

Research map / Optimization / Implicit bias and generalization

The Pitfalls of Simplicity Bias in Neural Networks

Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, Praneeth Netrapalli

NeurIPS 2020 · Optimization › Implicit bias and generalization · approach: Stability-based generalization analysis · 98 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Train an FCN on ^LMS-7, where the simplest feature is noisy and less predictive than the 7-slab featuresSGD-trained FCNs · ^LMS-7 testaccuracy: 0.940 ± 0.002 to 0.949 ± 0.003 across listed FCNs; text reports 95% test accuracyHurt
Randomize the simple linear featureFCN on LMS-5 · LMS-5randomized AUC: 1.0 to 0.5Hurt
Randomize all complex featuresFCN on LMS-5 · LMS-5randomized AUC: 1.00No clear effect
Train on MNIST-CIFAR and randomize the MNIST blockMobileNetV2 · MNIST-CIFAR:AAUC: 1.00 ± 0.00 to 0.53 ± 0.01Hurt
Randomize the CIFAR10 blockMobileNetV2 · MNIST-CIFAR:AAUC: 1.00 ± 0.00 to 1.00 ± 0.00No clear effect
Replace SGD with Adam or RMSProp, or change the activation among ReLU, Leaky ReLU, PReLU, and Tanh (ablation)(100,2)-FCN · LMS-7 and MS-(5,7)simple-feature-randomized AUC: approximately 0.50 across tested combinationsNo clear effect
Add dropout and vary ℓ2 regularization (ablation)FCNs on LMS-7 · LMS-7simple-feature-randomized AUC: approximately 0.5 for all modelsNo clear effect
Use a vanilla ensemble of independently trained models (ablation)Models on proposed datasets · LMS-5reliance on complex featuresNo clear effect
Use adversarial training (ablation)FCNs on proposed datasets · Synthetic datasetsreliance on complex featuresNo clear effect
Remove the simplest feature from the training data (ablation)SGD-trained models · LMS-5 and related synthetic dataaccuracy: 100%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy