Research map / Optimization / Implicit bias and generalization
Tunable Sensitivity to Large Errors in Neural Network Training
Gil Keren, Sivan Sabato, Björn Schuller
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a smaller k (0.125–0.5) rather than k=1 in single-layer networks | Single-layer feed-forward networks · MNIST, SVHN, CIFAR-10, and CIFAR-100 | test error: 1.76% -> 1.74%; 16.88% -> 16.16%; 48.32% -> 47.06%; 75.18% -> 74.41% | Improved |
| Use k=2 instead of k=1 | 3-layer network · SVHN, 400 units, momentum 0.5 | test error: 16.52% -> 16.52% | No clear effect |
| Use k=2 instead of k=1 | 3-layer network · CIFAR-10, 400 units, momentum 0.5 | test error: 46.81% -> 46.63% | Improved |
| Use k=0.5 instead of k=1 | 5-layer network · MNIST, 400 units, momentum 0.5 | test error: 1.71% -> 1.69% | Improved |
| Use k=4 instead of k=1 | 5-layer network · SVHN, 400 units, momentum 0.5 | test error: 17.41% -> 16.49% | Improved |
| Use k=4 instead of k=1 | 5-layer network · CIFAR-10, 800 units, momentum 0.5 | test error: 44.21% -> 44.24% | Hurt |
| Use k=4 instead of k=1 | 5-layer network · CIFAR-100, 800 units, momentum 0.9 | test error: 74.32% -> 74.62% | Hurt |
| Reduce sensitivity to large prediction errors using k=0.0625 | One-input, two-class synthetic classifier · Synthetic toy classification | test error: 4.90% -> 3.61% | Improved |
| Reduce k from 1 to 0.0625 | One-input, two-class synthetic classifier · Synthetic toy classification | test cross-entropy loss: 0.288 -> 1.190 | Hurt |
| Select k by cross-validation rather than fixing it at 1 | Feed-forward networks across the reported benchmarks · MNIST, SVHN, CIFAR-10, and CIFAR-100 | test error | Improved |
Closest papers by meaning
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.86)
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions (ICML 2025 · similarity 0.86)
- Explainable Neural Networks with Guarantee: A Sparse Estimation Approach (AAAI 2025 · similarity 0.86)
- Universality in Transfer Learning for Linear Models (NeurIPS 2024 · similarity 0.85)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.84)
- Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
