Research map / Learning theory and algorithms / Kernels and regression theory
Deep Equals Shallow for ReLU Networks in Kernel Regimes
Alberto Bietti, Francis Bach
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase ReLU random-feature network depth | Fully connected ReLU RF kernels · Sphere; asymptotic eigenvalue decay | eigenvalue decay exponent: k^{-d-2} | No clear effect |
| Increase ReLU network depth | Fully connected ReLU NTK kernels · Sphere; asymptotic eigenvalue decay | eigenvalue decay exponent: k^{-d} | No clear effect |
| Use step activations and compose the random-neuron kernel with depth | Step-activation random-neuron kernels · Sphere; asymptotic eigenvalue decay | eigenvalue decay exponent: ν = 1/2^{L-1} | Improved |
| Use shallow instead of three-layer random ReLU features | Random-feature regression · Synthetic f₂* | expected squared error | Improved |
| Increase RF kernel depth from 2 to 3 layers | RF kernel ridge regression · MNIST test | accuracy: 98.60 ± 0.03 -> 98.67 ± 0.03 | No clear effect |
| Increase RF kernel depth from 2 to 3 layers | RF kernel ridge regression · Fashion-MNIST test | accuracy: 90.75 ± 0.11 -> 90.87 ± 0.16 | No clear effect |
| Increase NTK depth from 2 to 3 layers | NTK kernel ridge regression · MNIST test | accuracy: 98.49 ± 0.02 -> 98.53 ± 0.02 | No clear effect |
| Increase NTK depth from 2 to 3 layers | NTK kernel ridge regression · Fashion-MNIST test | accuracy: 90.65 ± 0.07 -> 90.62 ± 0.08 | No clear effect |
| Use RF rather than NTK kernels | Kernel ridge regression · Fashion-MNIST test | accuracy: 90.87 ± 0.16 vs 90.62 ± 0.08 | Improved |
Closest papers by meaning
- Deep Network Approximation: Beyond ReLU to Diverse Activation Functions (JMLR 2024 · similarity 0.89)
- Diving into the shallows: a computational perspective on large-scale shallow learning (NeurIPS 2017 · similarity 0.89)
- Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers (ICLR 2022 · similarity 0.89)
- On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths (ICML 2021 · similarity 0.89)
- Understanding Deep Neural Networks with Rectified Linear Units (ICLR 2018 · similarity 0.88)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
