Research map / Architectures and efficiency / RNNs and normalization
Improving Memory Efficiency for Training KANs via Meta Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MetaKANs, which generate KAN spline/basis weights from learnable per-edge prompts via a small shared meta-learner MLP, cutting trainable parameters to roughly MLP scale while matching or beating KAN accuracy on function fitting, PDEs, and image classification.
Problem
KANs have many more trainable parameters than MLPs (roughly (G+k) times more because each edge carries a spline), causing high memory use and training cost that limit scalability.
Approach
Replace each edge's learnable spline coefficients with a learnable scalar prompt per activation function, and generate the coefficients with a small shared two-layer MLP meta-learner trained end-to-end; deep KANs use clustered layer-wise meta-learners. The same idea is applied to FastKAN, WavKAN, and ConvKAN variants.
Key results
- On Feynman functions at G=5, MetaKAN achieved lower MSE in 11 of 17 functions with 21% to 47% fewer trainable parameters; for I.12.5 MSE was 1.16e-4 vs 1.32e-3 with 30 vs 57 parameters.
- On MNIST, CIFAR-10 and CIFAR-100 convolutional 4-layer setups, MetaKANConv used about 392K parameters vs 3.49M for KANConv (roughly 89% reduction), with CIFAR-10 accuracy 45.97% vs 41.92%.
- On 8-layer MetaFastKANConv, CIFAR-10 accuracy reached 78.73% vs 79.84% for FastKANConv with about 78% fewer parameters (9.05M vs 40.7M); the authors note image classification may underperform existing baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate KAN activation weights using a meta-learner | KAN · Feynman functions, G=5 and G=20 | MSE: I.12.5, G=5: 1.32\times 10^{-3} -> 1.16\times 10^{-4}; I.14.3, G=20: 4.66\times 10^{-3} -> 5.73\times 10^{-3} | Mixed |
| Generate convolutional KAN activation weights using a meta-learner | KANConv · CIFAR-10, 4-layer convolutional model | accuracy: 41.92 -> 45.97 | Helped |
| Generate FastKAN convolutional activation weights using a meta-learner | FastKANConv · CIFAR-100, 4-layer convolutional model | accuracy: 34.64 -> 32.11 | Hurt |
| Generate FastKAN convolutional activation weights using a meta-learner and multiple meta-learners | FastKANConv · CIFAR-100, 8-layer convolutional model | accuracy: 50.80 -> 52.02 | Helped |
| Generate FastKAN activation weights using a meta-learner | FastKAN · CIFAR-100, hidden layers 32,32, c=13 | accuracy: 21.08 -> 23.17 | Helped |
| Generate KAN activation weights using a meta-learner | KAN · CIFAR-10, hidden layers 32,32, G=3 | accuracy: 42.16 -> 49.75 | Helped |
| Generate KAN activation weights using a meta-learner | KAN · High-dimensional f2, dimension 500 | MSE: 1.83e+2 -> 5.52e-2 | Helped |
| Use a shared meta-learner across deep KAN layers instead of layer-specific meta-learners | Deep KAN · Deep KAN experiments | performance | Hurt |
Surprising findings
- MetaKAN frequently achieved higher accuracy than the baseline KAN despite far fewer parameters, e.g. CIFAR-100 with 32 hidden units at G=3 (21.09% vs 4.45%).
- Under the paper's own parameter-count analysis, very small KANs can have more parameters with MetaKAN than with KAN because of the fixed meta-learner cost.
Limitations
- Image classification accuracy is often below existing KAN baselines and sometimes below MLP-like baselines, as the authors acknowledge.
- For very small KANs the fixed meta-learner cost can make MetaKAN have more parameters than KAN.
- Choosing meta-learner hidden width and prompt dimension depends on grid size and requires grid search.
Experimental setup
Datasets: Feynman symbolic regression dataset MNIST CIFAR-10 CIFAR-100 SVHN FMNIST KMNIST
Benchmarks: Feynman symbolic regression high-dimensional function fitting MNIST CIFAR-10 CIFAR-100 SVHN FMNIST KMNIST
Metrics: MSE accuracy trainable parameter count peak GPU memory
Baselines: KAN FastKAN WavKAN KANConv FastKANConv KALNConv KAGNConv MLP
Builds on: Kolmogorov-Arnold Networks FastKAN WavKAN ConvKAN HyperNetworks
Closest papers by meaning
- On the expressiveness and spectral bias of KANs (ICLR 2025 · similarity 0.88)
- KAN: Kolmogorov–Arnold Networks (ICLR 2025 · similarity 0.87)
- Dynamic Kernel Selection for Improved Generalization and Memory Efficiency in Meta-Learning (CVPR 2022 · similarity 0.86)
- PowerMLP: An Efficient Version of KAN (AAAI 2025 · similarity 0.86)
- Memory-Reduced Meta-Learning with Guaranteed Convergence (AAAI 2025 · similarity 0.85)
- Kolmogorov-Arnold Transformer (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.