Research map / Architectures and efficiency / Attention and transformers
PLASH: Provably Linear-Time Attention with Selective Higher-Order Feature Sketching
Yuwen Huang, Xiang Pan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PLASH M=16 instead of Standard FlashAttention-2 | Qwen3-4B · PG-19 N=16,384 | EMA training loss: 25.2 -> 23.5 | Improved |
| use PLASH M=32 instead of Standard at 32K | Qwen3-4B · PG-19 N=32,768 | EMA training loss: 40.1 -> 40.1 | No clear effect |
| increase prototypes M from 8 to 16 (ablation) | PLASH · PG-19 N=16,384 | EMA training loss: 30.1 -> 23.5 | Improved |
| increase prototypes M from 16 to 32 (ablation) | PLASH · PG-19 N=16,384 | EMA training loss: 23.5 -> 24.5 | Hurt |
| increase prototypes M from 8 to 32 (ablation) | PLASH · PG-19 N=32,768 | EMA training loss: 41.2 -> 40.1 | Improved |
| increase prototypes M from 8 to 32 (ablation) | PLASH · Forward time, N=32K | time (ms): 1809 -> 2536 | Hurt |
| use PLASH M=8 instead of Standard FlashAttention-2 | Qwen3-4B · Forward time, N=32K | time (ms): 4358 -> 1809 | Improved |
| use PLASH M=64 instead of Vanilla attention | Informer · Run time, length 11264 | latency (ms): 58.95 -> 2.69 | Improved |
| increase M from 64 to 512 (ablation) | PLASH · Run time, length 11264 | latency (ms): 2.69 -> 4.89 | Hurt |
| increase sketch size Dk from 128 to 512 at M=512 (ablation) | PLASH · ETTh2 (L=96) | MSE: 0.4700 -> 0.8952 | Hurt |
| increase M from 64 to 512 at Dk=128 (ablation) | PLASH · ETTh2 (L=96) | MSE: 1.0193 -> 0.4700 | Improved |
| use PLASH M=512 Dk=128 instead of Vanilla | Informer · WTH (L=96) | MSE: 0.3121 -> 0.3009 | Improved |
Closest papers by meaning
- MVA: Linear Attention with High-order Query-Keys Integration and Multi-level Vocabulary Decomposition (ICML 2025 · similarity 0.87)
- ZeroS: Zero‑Sum Linear Attention for Efficient Transformers (NeurIPS 2025 · similarity 0.87)
- Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.86)
- SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.86)
- Loki: Low-rank Keys for Efficient Sparse Attention (NeurIPS 2024 · similarity 0.86)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
