Research map / Architectures and efficiency / Pruning and efficient tuning
LoKi: Low-dimensional KAN for Efficient Fine-tuning Image Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoKi, an adapter-style PEFT module that places a 2-layer KAN as a learnable activation between linear down- and up-projections, evaluated for fine-tuning ViT-B/16 on small image classification and video action recognition datasets.
Problem
Adapter- and prompt-based PEFT methods, modeled as MLPs with fixed activations, overfit and generalize inconsistently across pre-trained weights and small downstream datasets with distribution shift.
Approach
Decouple nonlinearity from linear layers: use single-layer linear encoder/decoder with low bottleneck ratio and a minimal two-layer Kolmogorov-Arnold Network as the learnable activation, yielding fewer parameters than Adapter and avoiding KAN's memory cost.
Key results
- On image classification across seven datasets, LoKi attains the best average accuracy (e.g., 85.4 average on DDSM/FGVC-type shifts with the smallest cross-weight gap of 2.3) versus Adapter's 67.9 average, with about half of Adapter's parameters.
- On Diving48, LoKi(ViT) reaches 89.5% top-1 with 32x1x1 views using 6M tunable parameters, above AIM's reproduced accuracy at all three views.
- On HMDB51, LoKi(ViT) achieves 73.3% top-1, above ST-Adapter (65.9%) and AdaptFormer (63.7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoKi instead of Bias Tuning | ViT image fine-tuning · FGVC Aircraft | top-1 accuracy: 69.8 -> 71.6 | Helped |
| Use LoKi instead of Bias Tuning | ViT image fine-tuning · DDSM | top-1 accuracy: 67.4 -> 74.2 | Helped |
| Use LoKi instead of Bias Tuning | ViT image fine-tuning · CIFAR10 | top-1 accuracy: 98.4 -> 98.1 | Hurt |
| Use LoKi instead of AIM-ViT-B/16 | CLIP(ViT-B/16) video fine-tuning · Diving48 (16×1×1) | top-1 accuracy: 83.0 -> 85.7 | Helped |
| Use LoKi instead of VideoPrompt | ViT-B/16 video fine-tuning · HMDB51 | top-1 accuracy: 66.4 -> 73.3 | Helped |
| Add Space LoKi (ablation) | Frozen Space & MLP video model · Diving48 | top-1 accuracy: 23.2 -> 79.3 | Helped |
| Add Time LoKi (ablation) | Frozen Space & MLP + Space LoKi · Diving48 | top-1 accuracy: 79.3 -> 86.4 | Helped |
| Add MLP LoKi (ablation) | Space LoKi + Time LoKi video model · Diving48 | top-1 accuracy: 86.4 -> 87.6 | Helped |
| Reduce LoKi activation depth to the minimum-depth configuration (ablation) | LoKi video module · Diving48 | top-1 accuracy: 83.3 -> 87.6 | Helped |
| Increase encoder/decoder width to shape (768, [48, 97, 48], 768) (ablation) | LoKi video module · Diving48 | top-1 accuracy: 87.6 -> 89.5 | Helped |
| Increase encoder/decoder width further to shape (768, [64, 129, 64], 768) (ablation) | LoKi video module · Diving48 | top-1 accuracy: 89.5 -> 88.6 | Hurt |
| Increase activation width from shape (768, [16, 4, 16], 768) to (768, [16, 33, 16], 768) (ablation) | LoKi video module · Diving48 | top-1 accuracy: 82.9 -> 87.6 | Helped |
Surprising findings
- Vanilla KAN with similar parameter count consumed about 33GB memory, far more than other methods.
- On large datasets (K400, SSv2), LoKi underperforms Adapter (80.0 vs 83.9 on K400; 56.3 vs 66.4 on SSv2), which the authors attribute to overfitting being less of a concern with more data.
Limitations
- Underperforms Adapter on large-scale video datasets (K400, SSv2).
- Best accuracy on CLIP weights is less satisfactory per the authors; generalization gains rely on small datasets.
- Hyperparameters r1, r2 chosen via ablation on Diving48, with KAN spline settings G=4, K=3 taken from prior work.
Experimental setup
Datasets: CIFAR10 CIFAR100 Food101 MNIST DDSM DTD FGVC Aircraft HMDB51 Diving48 Kinetics-400 Something-Something v2
Benchmarks: CIFAR10 CIFAR100 Food101 MNIST DDSM DTD FGVC Aircraft HMDB51 Diving48 Kinetics-400 Something-Something v2
Metrics: top-1 accuracy absolute accuracy difference across pre-trained weights tunable parameters memory (GB)
Baselines: Full-tuning Linear Probe Bias Tuning Prompt Tuning LoRA Adapter KAN AIM AdaptFormer ST-Adapter VideoPrompt Pro-tuning
Builds on: Adapter LoRA Kolmogorov-Arnold Network AIM VPT
Closest papers by meaning
- Strong Baselines for Parameter-Efficient Few-Shot Fine-Tuning (AAAI 2024 · similarity 0.88)
- Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognition (CVPR 2025 · similarity 0.88)
- PELA: Learning Parameter-Efficient Models with Low-Rank Approximation (CVPR 2024 · similarity 0.87)
- SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors (NeurIPS 2024 · similarity 0.87)
- Sensitivity-Aware Efficient Fine-Tuning via Compact Dynamic-Rank Adaptation (CVPR 2025 · similarity 0.87)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.