Research map / Architectures and efficiency / Attention and transformers
LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel
Zhe Feng, Sen Lian, Changwei Wang, Muyang Zhang, Tianlong Tan, Rongtao Xu, Weiliang Meng, Xiaopeng Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Newton-Schulz iteration instead of conjugate gradient for kernel inverse (ablation) | LaplacianFormer-Tiny · ImageNet-1K | top-1 accuracy (%): 79.2 -> 81.4 | Improved |
| Use Newton-Schulz instead of CG (ablation) | LaplacianFormer-Small · ImageNet-1K | top-1 accuracy (%): 81.4 -> 83.8 | Improved |
| Set Laplacian kernel scale lambda to 4 instead of 0.5 (ablation) | LaplacianFormer-Tiny · ImageNet-1K | top-1 accuracy (%): 79.4 -> 81.4 | Improved |
| Increase kernel scale lambda from 4 to 8 (ablation) | LaplacianFormer-Tiny · ImageNet-1K | top-1 accuracy (%): 81.4 -> 78.5 | Hurt |
| Replace Gaussian kernel with Laplacian kernel in SOFT++ and Skyformer (ablation) | SOFT++, Skyformer · ImageNet-1K | convergence and accuracy | Improved |
| Use depthwise-convolution landmark selection instead of average pooling (ablation) | LaplacianFormer · ImageNet-1K | top-1 accuracy | No clear effect |
| Use LaplacianFormer-Medium instead of SOFT++-Medium | LaplacianFormer · ImageNet-1K | top-1 accuracy (%): 83.7 -> 85.3 | Improved |
| Use LaplacianFormer-Small instead of SOFT++-Small as detector backbone | LaplacianFormer · COCO Mask R-CNN 1x | box AP: 43.8 -> 45.8 | Improved |
| Use LaplacianFormer-Medium instead of SOFT++-Medium under RetinaNet | LaplacianFormer · COCO RetinaNet 1x | box AP: 44.3 -> 47.2 | Improved |
| Use Laplacian kernel CUDA kernels | LaplacianFormer · Kernel and solver runtime | execution time | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- SOFT++-Medium — SOFT: Softmax-free Transformer with Linear Complexity (NeurIPS 2021) · better on 12 of 12 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.90)
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.89)
- SOFT: Softmax-free Transformer with Linear Complexity (NeurIPS 2021 · similarity 0.88)
- Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials (NeurIPS 2025 · similarity 0.88)
- PolaFormer: Polarity-aware Linear Attention for Vision Transformers (ICLR 2025 · similarity 0.87)
- SeTformer Is What You Need for Vision and Language (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
