Research map / Architectures and efficiency / RNNs and normalization
Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes decomposing high-frequency functions into sum-compositions of low-frequency shallow networks, implemented via multi-grade deep learning (MGDL), which outperforms single-grade training (SGDL) on spectral-bias regression tasks.
Problem
Deep neural networks trained end-to-end (single-grade deep learning) exhibit spectral bias, learning low-frequency components well but struggling to capture high-frequency features of a function.
Approach
Represent a high-frequency function as a sum of compositions of low-frequency functions (motivated by the Jacobi–Anger identity) and learn it grade by grade with MGDL: each grade trains a shallow network on the residue of previous grades, composed with the frozen features from earlier grades.
Key results
- On synthetic 1D settings, MGDL reduces test RSE from about 1e-1 (SGDL) to about 1e-5 with similar or less training time, a 592 to 7,058x reduction across four settings.
- On Div2K/Cat images, MGDL outperforms SGDL by 2.35 to 3.93 dB test PSNR (e.g., Cat 24.18 dB vs 21.83 dB for SGDL).
- On modified MNIST with beta=1, MGDL improves test RSE by roughly 27-29% over SGDL across frequencies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MGDL instead of SGDL | Synthetic oscillatory regression · Synthetic settings 1–4 test | TeRSE: $1.2\times 10^{-1}$ -> $1.7\times 10^{-5}$; $5.7\times 10^{-3}$ -> $6.5\times 10^{-6}$; $1.1 \times 10^{-1}$ -> $2.1\tim | Helped |
| Use MGDL instead of SGDL | Manifold regression · Manifold settings 1–2, q=0 and q=4 test | TeRSE: $2.8\times 10^{-1}$ -> $1.1\times 10^{-4}$; $5.4 \times 10^{-1}$ -> $2.1\times 10^{-4}$; $2.3\times 10^{-4}$ -> $4.2\tim | Helped |
| Use MGDL instead of SGDL | Colored-image regression · Cat, Sea, and Building test images | PSNR: 2.35, 3.93 and 2.88 dB | Helped |
| Add MGDL grades sequentially (grade 1 through grade 4) (ablation) | MGDL · Cat, Sea, and Building test images | PSNR: Cat: $20.41$ -> $22.67$ -> $23.71$ -> $24.18$; Sea: $18.62$ -> $21.50$ -> $23.42$ -> $24.32$; Building: $17.29$ -> $18.9 | Helped |
| Use MGDL instead of SGDL | MNIST regression with beta=1 · MNIST modified data, kappa=1, 5, 10, and 50 test | TeRSE: about $27 \sim 29\%$ | Helped |
| Use MGDL instead of SGDL | MNIST regression · MNIST, beta=0.5 | validation loss | No clear effect |
| Increase the high-frequency noise amplitude beta | SGDL and MGDL · MNIST modified data | validation loss | Hurt |
Surprising findings
- On the modified MNIST task, MGDL has higher training RSE than SGDL but lower validation and test RSE, indicating SGDL overfits the high-frequency noise.
Limitations
- No theoretical foundation for why MGDL addresses spectral bias; mathematical understanding of spectral bias absent.
- Numerical studies are preliminary and limited to four examples.
- Only standard fully connected DNNs were tested, not CNNs or ResNets.
Experimental setup
Datasets: synthetic sinusoid mixtures flower-shaped manifold data Cat image Div2K (sea, building) MNIST
Benchmarks: MNIST (modified with radial wave noise) Div2K images synthetic sinusoid regression
Metrics: relative squared error (RSE) PSNR
Baselines: SGDL (single-grade deep learning) Fourier feature networks ResNet relay backpropagation
Builds on: multi-grade deep learning Jacobi-Anger identity relay backpropagation Fourier features
Closest papers by meaning
- Spectral Bias in Practice: The Role of Function Frequency in Generalization (NeurIPS 2022 · similarity 0.91)
- On the Spectral Bias of Neural Networks (ICML 2019 · similarity 0.90)
- Tuning Frequency Bias of State Space Models (ICLR 2025 · similarity 0.85)
- FINER: Flexible Spectral-bias Tuning in Implicit NEural Representation by Variable-periodic Activation Functions (CVPR 2024 · similarity 0.85)
- Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations (ICML 2025 · similarity 0.84)
- Spiking Neural Networks Need High-Frequency Information (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.