Research map / Architectures and efficiency / Attention and transformers
Kolmogorov-Arnold Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces transformer MLP layers with Group-Rational KAN (GR-KAN) layers that use CUDA-implemented rational base functions, grouped parameter sharing, and variance-preserving initialization, yielding KAT models that outperform ViT/DeiT on ImageNet-1K and transfer to detection and segmentation.
Problem
Kolmogorov-Arnold Networks (KANs) do not scale to large transformers: B-spline bases are GPU-unfriendly, parameters and FLOPs grow with (G+K) per input-output edge, and default initialization breaks variance preservation, so naive ViT+KAN fails on ImageNet.
Approach
Swap B-spline bases for Horner-evaluated rational functions implemented in CUDA, share rational parameters across groups of input channels (keeping per-edge scalar weights), and initialize rational coefficients to fit identity/Swish with a numerically estimated gain so activation variance is preserved; use this GR-KAN as the MLP replacement in ViT.
Key results
- KAT-B reaches 82.3% top-1 on ImageNet-1K versus 81.8% for DeiT-B and 79.1% for ViT-B at the same 86.6M parameters; KAT-B* (ViT-initialized) reaches 82.8%.
- Naive ViT+KAN reaches only 64.9% (Tiny) and 62.9% (Small), and ViT-B+KAN diverges (NaN).
- KATDet-S improves box AP to 47.5 vs 44.5 for ViTDet-S on COCO Mask R-CNN 3x schedule.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the MLP channel mixer with GR-KAN | ViT-S/16 · ImageNet-1K val | Top-1 accuracy: 81.2 vs 79.8; 2.4% | Helped |
| Replace the MLP channel mixer with GR-KAN | ViT-Ti/16, ViT-S/16, and ViT-B/16 · ImageNet-1K val | Top-1 accuracy: 72.7 -> 74.6; 78.8 -> 81.2; 79.1 -> 82.3 | Helped |
| Initialize KAT from pre-trained ViT weights | KAT · ImageNet-1K val | Top-1 accuracy: 74.6 -> 75.7; 81.2 -> 82.0; 82.3 -> 82.8 | Helped |
| Replace MLP with standard KAN | ViT-T/S · ImageNet-1K val | Top-1 accuracy: 72.7 -> 64.9; 78.8 -> 62.9 | Hurt |
| Replace ViTDet-S backbone with KATDet-S | Mask R-CNN · MS-COCO2017 val | APbox: 44.5 -> 47.5; 3.0 APbox gain | Helped |
| Replace ViTDet-B backbone with KATDet-B | Mask R-CNN · MS-COCO2017 val | APbox: 46.3 -> 47.7; 1.4 APbox gain | Helped |
| Replace DeiT-S with KAT-S as UperNet backbone | UperNet · ADE20K val | mIoU: 43.5 -> 46.1; 2.4% | Helped |
| Replace DeiT-B with KAT-B as UperNet backbone | UperNet · ADE20K val | mIoU: 47.2 -> 47.4; 0.2% | Helped |
| Use KAT-T with learnable group-rational activations instead of GELU (ablation) | ViT-Ti/16 · ImageNet-1K val | Top-1 accuracy: 72.7 -> 74.6; 1.9% | Helped |
| Replace GELU with ReLU (ablation) | ViT-Ti/16 · ImageNet-1K val | Top-1 accuracy: 72.7 -> 72.8 | No clear effect |
| Replace GELU with SiLU (ablation) | ViT-Ti/16 · ImageNet-1K val | Top-1 accuracy: 72.7 -> 69.8 | Hurt |
| Replace GELU with learnable PReLU (ablation) | ViT-Ti/16 · ImageNet-1K val | Top-1 accuracy: 72.7 -> 73.2 | Helped |
Surprising findings
- The rational-function KAT is slower than the ReLU/GELU/SiLU baselines in throughput (2313 vs ~2650 batch/s on A5000), and the authors note gradient instability for the rational coefficients at higher orders.
- A plain SiLU MLP activation scores 69.8% vs GELU's 72.7%, while learnable PAU/PReLU give modest gains; the KAT gain is not attributable to the learnable activation alone.
Limitations
- Rational function evaluation is slower than standard activations, reducing throughput.
- Higher-order gradients of rational coefficients can be unstable.
- Evaluated only on vision tasks; not tested on language modeling.
Experimental setup
Datasets: ImageNet-1K MS-COCO 2017 ADE20K
Benchmarks: ImageNet-1K MS-COCO 2017 ADE20K
Metrics: top-1 accuracy AP box AP mask mIoU FLOPs throughput peak memory
Baselines: ViT DeiT ViT+KAN Swin Transformer ConvNeXt ViTDet PVT
Builds on: Kolmogorov-Arnold Networks ViT DeiT Padé activation units
Closest papers by meaning
- KAN: Kolmogorov–Arnold Networks (ICLR 2025 · similarity 0.90)
- On the expressiveness and spectral bias of KANs (ICLR 2025 · similarity 0.88)
- PowerMLP: An Efficient Version of KAN (AAAI 2025 · similarity 0.87)
- Incorporating Arbitrary Matrix Group Equivariance into KANs (ICML 2025 · similarity 0.87)
- Generalization Bounds and Model Complexity for Kolmogorov–Arnold Networks (ICLR 2025 · similarity 0.86)
- Equivariant Neural Functional Networks for Transformers (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.