Research map / Architectures and efficiency / RNNs and normalization
PowerMLP: An Efficient Version of KAN
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PowerMLP, an MLP-type network using powers of ReLU plus a basis function to replace KAN's recursive B-spline computation, claiming equal function space on bounded intervals and ~40x faster training than KAN.
Problem
KAN is much slower to train and run than MLP because its spline activations require iterative de Boor-Cox B-spline recursion, limiting its practical use.
Approach
Represent B-splines as linear combinations of powers of ReLU (non-iterative), and place the resulting ReLU-k activations on nodes (as in an MLP) with a residual basis function, yielding PowerMLP which is theoretically a superset of KAN over R^n and equal over bounded intervals, with far lower FLOPs.
Key results
- PowerMLP achieves best test RMSE on 11/16 small-size and 10/16 large-size special function fitting cases versus KAN and MLP.
- Training time averaged over 8 tasks is about 40x faster for PowerMLP than KAN (e.g., JE: 0.738 s vs 17.73 s on KAN).
- Theory: FLOPs ratio of KAN layer to PowerMLP layer is over 10x; PowerMLP defines a strictly larger function space than KAN over R^n and equal over bounded intervals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace KAN spline activations with PowerMLP's ReLU-k activations and basis function | KAN · JE function fitting, small and large sizes | test RMSE: $4.63\times10^{-3}$ -> $\mathbf{5.79\times 10^{-4}}$; $1.04\times10^{-4}$ -> $\mathbf{7.23\times 10^{-5}}$ | Helped |
| Replace KAN with PowerMLP | KAN · Function fitting across 16 special functions | test RMSE | Mixed |
| Replace KAN with PowerMLP | KAN · Knot theory, geometric-invariant input cases | test accuracy | Helped |
| Replace KAN with PowerMLP | KAN · Titanic, Income, Spam, AG_NEWS, MNIST, and SVHN | test accuracy | Helped |
| Replace MLP with PowerMLP | MLP · Income, Titanic, AG_NEWS, Spam, SVHN, and MNIST | test accuracy | Mixed |
| Replace KAN with PowerMLP | KAN · Training time across eight tasks | training time | Helped |
| Remove the basis function (ablation) | PowerMLP · f(x,y)=x exp(-y), varying network depth | test RMSE | Hurt |
| Remove the basis function (ablation) | PowerMLP · f(x,y)=x exp(-y), varying parameter count | test RMSE | Mixed |
| Replace ReLU-k activation with ReLU (ablation) | PowerMLP · f(x,y)=x exp(-y), across tested settings | test RMSE | Hurt |
Surprising findings
- PowerMLP underperforms MLP on image classification (SVHN, MNIST), per the authors' own summary.
- In a supplementary CNN experiment on CIFAR-10, 5-order PowerMLP failed to converge across all seeds tried.
Limitations
- Fails to achieve exceptional performance on complicated computer vision tasks and long-text processing.
- Lacks specialized structures such as convolution or attention; only MLP-level architecture studied.
- Higher-order ReLU powers can fail to converge (5-order in CNN experiment).
Experimental setup
Datasets: 16 special functions (KAN benchmark) knot theory invariants (signature dataset) Titanic Income SMS Spam Collection AG_NEWS MNIST SVHN CIFAR-10 CoLA
Benchmarks: special function fitting (16 functions) knot theory signature prediction Titanic Income Spam AG_NEWS MNIST SVHN CIFAR-10 CoLA
Metrics: test RMSE test accuracy training time (s) FLOPs
Baselines: KAN MLP
Builds on: Kolmogorov-Arnold Network ReLU-k / RePU activations de Boor-Cox formula for B-splines
Closest papers by meaning
- On the expressiveness and spectral bias of KANs (ICLR 2025 · similarity 0.92)
- KAN: Kolmogorov–Arnold Networks (ICLR 2025 · similarity 0.91)
- Generalization Bounds and Model Complexity for Kolmogorov–Arnold Networks (ICLR 2025 · similarity 0.87)
- Kolmogorov-Arnold Transformer (ICLR 2025 · similarity 0.87)
- Incorporating Arbitrary Matrix Group Equivariance into KANs (ICML 2025 · similarity 0.86)
- Improving Memory Efficiency for Training KANs via Meta Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.