Research map / Optimization / Implicit bias and generalization
Assessing Generalization of SGD via Disagreement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use two runs with different data and initialization instead of only different initialization | ResNet18 · CIFAR-10 (10000 training examples) | disagreement: 0.307 -> 0.348 | Mixed |
| Use two runs with different data order instead of different initialization | ResNet18 · CIFAR-10 (10000 training examples) | gap: 0.030 -> 0.033 | Mixed |
| Vary training data or data and initialization rather than only initialization or data order | ResNet18 · CIFAR-10 (10000 training examples) | CACE (100 models): 0.0979 -> 0.0437; 0.1014 -> 0.0491 | Helped |
| Estimate test error with pairwise disagreement between same-dataset SGD runs | ResNet18 · CIFAR-10 (10000 training examples) | test error versus disagreement gap: 0.012; 0.013; 0.030; 0.033 | Mixed |
| Use ImageNet pre-training rather than random initialization | ResNet50 · PACS distribution shifts | GDE agreement | Helped |
| Evaluate on target domains outside the training domain | ResNet50 · PACS distribution shifts | GDE agreement | Mixed |
Closest papers by meaning
- (Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancy (NeurIPS 2023 · similarity 0.87)
- Inconsistency, Instability, and Generalization Gap of Deep Neural Network Training (NeurIPS 2023 · similarity 0.87)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.86)
- Learning Curves for SGD on Structured Features (ICLR 2022 · similarity 0.86)
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima (ICLR 2017 · similarity 0.86)
- Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.