Research map / Optimization / Implicit bias and generalization
Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures
Emmanuel Abbe, Samy Bengio, Elisabetta Cornacchia, Jon Kleinberg, Aryo Lotfi, Maithra Raghu, Chiyuan Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase depth of linear neural network (holdout generalization error vs Boolean influence) | linear neural network, hidden size 256 · linear target function on 11 bits, canonical holdout | generalization error | Improved |
| decrease initialization scale (alpha from 0.5 to 2.5) in 3-layer linear network | 3-layer linear neural network · linear target function on 11 bits, canonical holdout | generalization error: alpha in {0.5, 1, 1.5, 2, 2.5} | Improved |
| use MLP-Mixer instead of MLP or Transformer | canonical holdout PVR learning · PVR with 3 pointer bits, window size 3, majority vote | gap between generalization error and Boolean influence | Hurt |
| use MLP or Transformer architecture | canonical holdout PVR learning · PVR with 3 pointer bits, window size 3, majority vote | gap between generalization error and Boolean influence | Improved |
| nonzero bias in the frozen linear target (hat f(empty) differs from hat f({k})) | linear regression with GD · canonical holdout, frozen coordinate k | expected generalization error | Hurt |
| minimum l2-norm interpolating solution instead of low-degree representation | 4-bit toy function with x2 frozen to 1 · canonical holdout example | generalization error: half the error of f_{-2} | Improved |
Closest papers by meaning
- Generalization on the Unseen, Logic Reasoning and Degree Curriculum (ICML 2023 · similarity 0.85)
- Neural Reasoning Networks: Efficient Interpretable Neural Networks with Automatic Textual Explanations (AAAI 2025 · similarity 0.83)
- Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.83)
- Learning High-Degree Parities: The Crucial Role of the Initialization (ICLR 2025 · similarity 0.82)
- The Gradient of Algebraic Model Counting (AAAI 2025 · similarity 0.82)
- Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
