Research map / Architectures and efficiency / Attention and transformers
LASER: Attention with Exponential Transformation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LASER attention, which applies softmax attention to exp-transformed values and takes a log (with a log-weighted-sum-exp shift for numerical stability), claiming larger softmax Jacobians and gains across LLMs up to 7.7B, ViT, Conformer and BERT.
Problem
Gradients backpropagated through the softmax in attention can be small when attention probabilities are small or saturated, leading to inefficient learning of parameters preceding attention.
Approach
Compute attention on exponentially transformed values, log(softmax(QK^T) exp(V)), which admits a larger Jacobian and a log-sum-exp (soft-max-like) structure; implement it by shifting values by their column max and wrapping any standard attention kernel (e.g., FlashAttention) with exp before and log after.
Key results
- On a 2.2B decoder-only LLM, LASER averages 63.39% one-shot accuracy vs 62.34% for standard attention (about 1% gain).
- ViT-S/16 on ImageNet: LASER validation error 0.2417 vs standard 0.2532 (about 1.15% absolute improvement).
- On 7.7B models trained on 44B tokens, LASER averages 53.97% vs 52.53% for standard attention on downstream tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace standard attention with LASER | 301M decoder-only Transformer · C4 test loss across the reported layer/hidden-dimension settings | test loss: 2.681 -> 2.673; 2.641 -> 2.595; 2.575 -> 2.555 | Helped |
| use LAMB optimizer with LASER attention instead of standard attention (ablation) | 301M decoder-only Transformer · C4 training and test loss | loss: training: 2.749 vs 2.736; test: 2.758 vs 2.741 | Helped |
| replace standard attention with LASER | 2.2B autoregressive Transformer · One-shot downstream task average | accuracy: 62.34% -> 63.39% | Helped |
| replace standard attention with LASER | 2.2B autoregressive Transformer · SuperGLUE fine-tuning average | decoding accuracy: 42.35 -> 44.01 | Helped |
| replace standard attention with LASER | 2.2B BERT · MLPerf masked language modeling | prediction error rate: 0.2145 -> 0.2125 | Helped |
| replace standard attention with LASER | ViT-S/16 · ImageNet-1k validation | error rate: 0.2532 -> 0.2417 | Helped |
| add temperature tuning to LASER (ablation) | ViT-S/16 · ImageNet-1k validation | error rate: 0.2417 -> 0.2427 | Hurt |
| add QK-normalization to LASER (ablation) | ViT-S/16 · ImageNet-1k validation | error rate: 0.2417 -> 0.2372 | Helped |
| add per-dimension temperature to LASER (ablation) | ViT-S/16 · ImageNet-1k validation | error rate: 0.2417 -> 0.2475 | Hurt |
| replace standard attention with LASER | Conformer · Librispeech validation | WER: 0.083192 -> 0.082892 | Helped |
| replace standard attention with LASER | 7.7B autoregressive Transformer · Downstream task average | accuracy: 52.53 -> 53.97 | Helped |
| add LASER to DiffTransformer attention (ablation) | 2.2B DiffTransformer · Downstream task average | accuracy: 50.59 -> 51.52 | Helped |
Surprising findings
- LASER gains persisted under LAMB optimizer normalization, suggesting gains are not solely due to larger gradient norms.
- Per-dimension temperature scaling hurt ViT performance for both attention variants.
Limitations
- Gains are small (about 1%) and often within reported standard deviations on downstream tasks.
- Exponential transform can overflow, requiring the log-weighted-sum-exp shift; the approximation to max relies on limiting-case analysis.
Experimental setup
Datasets: C4 ImageNet-1k LibriSpeech MLPerf Wikipedia BERT data SuperGLUE
Benchmarks: ARC BoolQ CB COPA HellaSwag MultiRC OpenBookQA PIQA RACE ReCoRD RTE StoryCloze
Metrics: test loss one-shot accuracy finetuning accuracy error rate word error rate training stability (loss spikes)
Baselines: Standard attention Standard+temp Standard+per-dim temp Standard+QK-normalization DiffTransformer
Builds on: Transformer FlashAttention Differential Transformer QK-normalization AlgoPerf
Closest papers by meaning
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.89)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.88)
- From Attention to Activation: Unraveling the Enigmas of Large Language Models (ICLR 2025 · similarity 0.88)
- Forgetting Transformer: Softmax Attention with a Forget Gate (ICLR 2025 · similarity 0.87)
- Memorization Capacity of Multi-Head Attention in Transformers (ICLR 2024 · similarity 0.87)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.