Research map / Architectures and efficiency / Attention and transformers
Implicit Kernel Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the fixed RBF similarity with a data-adaptive implicit kernel | Transformer attention · Sentence classification: MR, CR, SST, SUBJ, TREC | accuracy: Transformer -> IKAN: 74.45$\pm$0.94\% -> 76.63$\pm$0.89\% (MR); 77.98$\pm$3.16\% -> 79.21$\pm$3.05\% (CR); 76.25$\pm$1.6 | Helped |
| Adapt the magnitude term using an L^p norm rather than fixed L^2 (ablation) | IKA_NS · Sentence classification | accuracy: IKA_NS -> IKAN: 76.42$\pm$0.82\% -> 76.63$\pm$0.89\% (MR); 79.05$\pm$2.26\% -> 79.21$\pm$3.05\% (CR); 77.49$\pm$1.84\% - | Helped |
| Learn spectral points directly without sampling (ablation) | IKAN · Sentence classification | accuracy: IKAN -> IKAN-direct: 76.63$\pm$0.89\% -> 75.97$\pm$0.54\% (MR); 79.21$\pm$3.05\% -> 79.26$\pm$3.05\% (CR); 77.49$\pm$1.8 | Hurt |
| Add copula-based dependency between attention heads (ablation) | IKAN · Sentence classification | accuracy: IKAN -> MIKAN: 76.63$\pm$0.89\% -> 76.70$\pm$1.03\% (MR); 79.21$\pm$3.05\% -> 80.63$\pm$3.24\% (CR); 77.49$\pm$1.84\% -> | Mixed |
| Replace Transformer attention with IKAN-direct | Transformer · IWSLT14 De-En | BLEU: 34.44 $\pm$ 0.07 -> 34.59 $\pm$ 0.09 | Helped |
| Add copula-structured multi-head attention (ablation) | IKAN-direct · IWSLT14 De-En | BLEU: 34.59 $\pm$ 0.09 -> \textbf{34.70} $\pm$ 0.09 | Helped |
| Replace Transformer attention with IKAN-direct | Transformer · UCI regression: CO2, Passenger, Housing, Concrete, Parkinsons | RMSE: Transformer -> IKAN-direct: 0.057 -> 0.055 (CO2); 0.153 -> 0.110 (Passenger); 0.126 -> 0.118 (Housing); 0.120 -> 0.111 ( | Helped |
| Add copula-structured multi-head attention (ablation) | IKAN-direct · UCI regression: CO2, Passenger, Housing, Concrete, Parkinsons | RMSE: IKAN-direct -> MIKAN: 0.055 -> 0.055 (CO2); 0.110 -> 0.142 (Passenger); 0.118 -> 0.108 (Housing); 0.111 -> 0.108 (Concre | Mixed |
| Replace GAT attention with IKAN-direct | GAT · Node classification: Cora, Citeseer, Pubmed | accuracy: GAT -> IKAN-direct: 83.0$\pm$0.7\% -> 83.3$\pm$0.7\% (Cora); 72.5$\pm$0.7\% -> 72.8$\pm$1.0\% (Citeseer); 79.0$\pm$ 0.3\ | Helped |
| Add copula-structured multi-head attention (ablation) | IKAN-direct · Node classification: Cora, Citeseer, Pubmed | accuracy: IKAN-direct -> MIKAN: 83.3$\pm$0.7\% -> \textbf{83.4}$\pm$0.6\% (Cora); 72.8$\pm$1.0\% -> \textbf{72.9}$\pm$0.8\% (Cites | Mixed |
| Reduce p toward zero in the magnitude norm (ablation) | MIKAN · Attention weights | sparsity | Helped |
Closest papers by meaning
- Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.87)
- DHAKR: Learning Deep Hierarchical Attention-Based Kernelized Representations for Graph Classification (AAAI 2025 · similarity 0.87)
- Graph Attention is Not Always Beneficial: A Theoretical Analysis of Graph Attention Mechanisms via Contextual Stochastic Block Models (ICML 2025 · similarity 0.86)
- Graph External Attention Enhanced Transformer (ICML 2024 · similarity 0.86)
- What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding (ICML 2024 · similarity 0.86)
- Graph Convolutions Enrich the Self-Attention in Transformers! (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.