Research map / Architectures and efficiency / RNNs and normalization
On the expressiveness and spectral bias of KANs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares KANs and MLPs theoretically (representation via ReLU^k equivalence, Sobolev approximation rates) and shows via a single-layer analysis and experiments that KANs with B-splines exhibit much less spectral bias than MLPs.
Problem
Whether Kolmogorov-Arnold Networks (KANs) are more expressive than MLPs and whether they suffer less from spectral bias, which matters for scientific computing tasks that require learning high frequencies.
Approach
Shows that ReLU^k MLPs can be exactly reparameterized as spline KANs of comparable size and that KANs (without SiLU) can be written as MLPs with width grown by grid size; analyzes the Hessian of a single-layer KAN least-squares problem to show a bounded condition number independent of grid size, and tests spectral bias empirically.
Key results
- KANs learn all frequencies of the 1D wave target at roughly the same rate once grid size and depth are large enough, whereas MLPs, even with 10x more epochs, struggle with high frequencies.
- For the 1D Poisson problem with k=16 and 32, MLP optimization becomes extremely hard while KAN residuals barely deteriorate with frequency.
- Theory: the eigenvalue ratio of the single-layer KAN least-squares Hessian is bounded by a constant depending only on spline degree k (independent of grid size), versus condition number scaling like n^4 for two-layer ReLU MLPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KAN instead of a ReLU MLP | 1D frequency fitting · 1D frequency fitting | high-frequency fitting | Helped |
| Use KAN instead of MLP on rough functions | Gaussian random field regression · Gaussian random field, small σ | training loss | Helped |
| Use KAN instead of MLP on smooth functions | Gaussian random field regression · Gaussian random field, large σ | training loss | Hurt |
| Use grid extension (ablation) | KAN · Gaussian random field regression | regression loss | Helped |
| Increase the number of training samples by 10x (ablation) | KAN regression · Gaussian random field, complicated task | overfitting: 10x | Helped |
| Use KAN instead of MLP | 1D Poisson solver · 1D Poisson equation, high-frequency solutions | relative L2 and H1 losses | Helped |
| Use KAN instead of MLP | 2D Poisson solver · 2D Poisson equation, high-frequency solutions | loss | Helped |
| Coarsen the spline grid (ablation) | KAN · Spectral bias and overfitting | spectral bias | Helped |
| Use a ReLU^k MLP instead of a standard MLP | Gaussian random field regression · Gaussian random field | training loss | Hurt |
Surprising findings
- ReLU^k MLPs performed even worse than standard MLPs on the Gaussian random field regression task.
- KANs' reduced spectral bias makes them more prone to overfitting noisy regression data, requiring more training points to mitigate.
Limitations
- Spectral bias analysis is restricted to a single KAN layer with continuous least-squares loss and is described as highly simplified.
- Unknown whether the KAN-to-MLP construction is sharp for deep networks.
- Current grid extension implementation is prone to oscillation in undersampled regimes.
Experimental setup
Datasets: synthetic 1D multi-frequency waves Gaussian random field samples 1D and 2D Poisson problems
Benchmarks: 1D multi-frequency wave regression Gaussian random field regression 1D Poisson equation (Deep Ritz) 2D Poisson equation (Deep Ritz)
Metrics: normalized DFT magnitude over training training loss test loss relative L2 loss relative H1 loss
Baselines: ReLU MLP tanh MLP ReLU^k MLP
Builds on: KAN Kolmogorov-Arnold representation theorem Deep Ritz method neural tangent kernel
Closest papers by meaning
- KAN: Kolmogorov–Arnold Networks (ICLR 2025 · similarity 0.95)
- PowerMLP: An Efficient Version of KAN (AAAI 2025 · similarity 0.92)
- Generalization Bounds and Model Complexity for Kolmogorov–Arnold Networks (ICLR 2025 · similarity 0.89)
- Kolmogorov-Arnold Transformer (ICLR 2025 · similarity 0.88)
- Incorporating Arbitrary Matrix Group Equivariance into KANs (ICML 2025 · similarity 0.88)
- Improving Memory Efficiency for Training KANs via Meta Learning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.