Research map / Optimization / Implicit bias and generalization
Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a 3-layer ReLU-like FNN with two hidden layers satisfying the stated width-product condition | ReLU-like FNN · Arbitrary dataset with distinct inputs | finite-sample expressivity: d_1 d_2 = Ω(N d_y) | Helped |
| Add a hidden layer for classification | ReLU-like FNN · Arbitrary classification dataset | hidden nodes required for memorization: 4√N + 4d_y | Helped |
| Use two hidden layers rather than one | ReLU FNN · Arbitrary scalar regression dataset | hidden nodes required for memorization: N -> 4√N | Helped |
| Use a deeper and/or narrower FNN with disjoint adjacent-layer pairs satisfying the stated width-product condition | Hard-tanh FNN · Arbitrary dataset with distinct inputs | finite-sample expressivity: 4 Σ floor((d_lj-r_j)/2) floor((d_lj+1-r_j)/(2d_y)) ≥ N | Helped |
| Use a ReLU-like ResNet under the general-position assumption | ReLU-like ResNet · Arbitrary classification dataset in general position | hidden nodes sufficient for memorization: 4N/d_x + 4d_y | Helped |
| Use a one-hidden-layer ReLU-like FNN under the general-position assumption | ReLU-like FNN · Arbitrary classification dataset in general position | hidden nodes sufficient for memorization: 4N/d_x + 4d_y | Helped |
| Initialize without-replacement SGD sufficiently close to a differentiable memorizing global minimum and use a sufficiently small step size | SGD on a piecewise-linear FNN · Empirical risk near a memorizing global minimum | empirical-risk gap at the first specified stopping iterate: O(‖ξ^(0)‖^2) -> C‖ξ^(0)‖^4 | Helped |
Closest papers by meaning
- On the Optimal Memorization Power of ReLU Neural Networks (ICLR 2022 · similarity 0.94)
- Network size and size of the weights in memorization with two-layers neural networks (NeurIPS 2020 · similarity 0.91)
- The Cost of Robustness: Tighter Bounds on Parameter Complexity for Robust Memorization in ReLU Nets (NeurIPS 2025 · similarity 0.91)
- An Exponential Improvement on the Memorization Capacity of Deep Threshold Networks (NeurIPS 2021 · similarity 0.90)
- Memorization Capacity of Neural Networks with Conditional Computation (ICLR 2023 · similarity 0.90)
- Memorization in Graph Neural Networks (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.