Research map / Architectures and efficiency / Attention and transformers

The Kernel Reality Check: Benchmarking and Distilling Efficient Attention at Scale

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Linear attention approximations

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Benchmarks kernelized (linear) attention methods at foundation-model scale and distills efficient attention variants, analyzing limitations on both the efficiency and quality sides.

Problem

Kernelized attention aims to replace O(N^2) softmax attention with O(N) alternatives, but systematic evaluations at foundation-model scale and fair cross-method comparisons are missing.

Approach

Analyzes kernelized attention in two halves (efficiency and quality), benchmarks approaches under a fair comparison at scale, and uses distillation to build efficient attention variants; only the abstract was available, so specifics are not stated.

Closest papers by meaning

• Sparse Attention as Compact Kernel Regression (NeurIPS 2026 · similarity 0.91)
• Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level (NeurIPS 2024 · similarity 0.89)
• Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
• The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry (ICLR 2024 · similarity 0.88)
• Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.87)
• Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.87)