Research map / Learning theory and algorithms / Kernels and regression theory
On the Spectral Bias of Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train by gradient descent on mixtures of sinusoids | 6-layer, 256-unit ReLU network · Synthetic sinusoid regression | frequency learning order | Helped |
| Use larger amplitudes for higher-frequency target components (ablation) | 6-layer, 256-unit ReLU network · Synthetic sinusoid regression | frequency learning order | No clear effect |
| Add low-frequency noise to the training target | 6-layer, 256-unit ReLU network · MNIST two-class target | best validation loss | Hurt |
| Add high-frequency noise to the training target (ablation) | 6-layer, 256-unit ReLU network · MNIST two-class target | best validation loss | No clear effect |
| Add high-frequency noise to the training target (ablation) | 6-layer, 256-unit ReLU network · MNIST target with kernel-eigenfunction noise | validation loss | Mixed |
| Increase the petal count of the data manifold (ablation) | ReLU network on γL · Sinusoid regression on γL | high-frequency learning speed | Helped |
| Increase the petal count of the data manifold (ablation) | ReLU network on γL · Binarized sinusoid classification on γL | classification accuracy | Helped |
| Increase network depth while holding width fixed (ablation) | ReLU network · Delta-function regression | high-frequency fitting ability | Helped |
| Increase network width while holding depth fixed (ablation) | ReLU network · Delta-function regression | high-frequency fitting ability | Helped |
| Increase the weight clip (ablation) | ReLU network · Delta-function regression | high-frequency fitting ability | Helped |
| Constrain the weight clip (ablation) | ReLU network · Delta-function regression | high-frequency fitting ability | Hurt |
| Use a complex MNIST reconstruction manifold instead of a flat manifold of the same dimension (ablation) | Identical ReLU networks · White-noise regression on MNIST reconstructions versus flat 64-dimensional inputs | training loss | Helped |
Closest papers by meaning
- The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies (NeurIPS 2019 · similarity 0.92)
- Spectral Bias in Practice: The Role of Function Frequency in Generalization (NeurIPS 2022 · similarity 0.92)
- Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning (NeurIPS 2024 · similarity 0.90)
- Simplicity Bias via Global Convergence of Sharpness Minimization (ICML 2024 · similarity 0.87)
- Tuning Frequency Bias of State Space Models (ICLR 2025 · similarity 0.86)
- Asymptotics of Learning with Deep Structured (Random) Features (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.