Research map / Architectures and efficiency / Attention and transformers
Viewing Transformers Through the Lens of Long Convolutions Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LaS-Attention, a transformer attention variant that adds an exponentially locally decaying mask and row-wise average-pool smoothing to attention, improving Long Range Arena performance without extra parameters.
Problem
Transformers perform poorly on long-range tasks (e.g., Long Range Arena) compared with state-space and global convolution layers; the paper asks which inductive biases explain this gap.
Approach
Identify smoothness and exponential locality as key inductive biases of long-range layers and inject them into attention: an exponentially decaying distance mask (ELD) multiplies attention scores per head, and 1-D average pooling over each attention row smooths them, adding no learnable parameters.
Key results
- LaS-Attention reaches 73.99 average on LRA versus 54.39 for the vanilla Transformer and 61.95 for Luna-256; LaS-Chunk averages 65.73.
- On sMNIST LaS improves from 98.90% to 99.18%, and on pMNIST from 97.90% to 98.05%.
- On the Select atomic task LaS-Attention scores R2 0.783 versus 0.971 for vanilla attention, showing it lags on non-local, non-smooth tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LaS smoothness and locality to attention | Transformer · LRA benchmark | average accuracy: +12.04% | Helped |
| Use LaS-Chunk attention with blocks of size 128 | Transformer · LRA benchmark | average accuracy: +11.34% | Helped |
| Add local and smooth operators | Transformer · sMNIST | accuracy: 98.90% -> 99.18% | Helped |
| Add local and smooth operators | Transformer · pMNIST | accuracy: 97.90% -> 98.05% | Helped |
| Add smoothness operator (ablation) | LaS-Chunk · LRA ListOps, Text, Retrieval, and Image | accuracy: 0.08%, 3.25%, 3.2%, and 3.52% | Helped |
| Add ELD locality operator (ablation) | LaS-Chunk · LRA ListOps, Text, Retrieval, and Image | accuracy: 5.13%, 8.46%, 2.42%, and 5.42% | Helped |
| Replace ELD with Alibi (ablation) | LaS-Chunk · LRA ListOps, Text, Retrieval, and Image | accuracy | Hurt |
| Narrow the context window by decreasing chunk size (ablation) | LaS transformer · LRA Image, Text, ListOps, and Pathfinder | accuracy | Hurt |
| Increase the number of training samples (ablation) | LaS transformer · LRA Image, Text, ListOps, and Pathfinder | accuracy | Helped |
| Add LaS inductive bias | Vanilla Attention · Select | R2 score: 0.971 -> 0.783 | Hurt |
| Add LaS inductive bias | Vanilla Attention · CumMax and CumSum | R2 score: 1.0 -> 0.995; 1.0 -> 0.997 | No clear effect |
| Add LaS attention | Vanilla Attention · Text8 | bits per byte: 1.3557 -> 1.3332 | Helped |
Surprising findings
- LaS-Attention underperforms vanilla attention on non-local, non-smooth atomic tasks such as Select.
- Adding the ALiBi bias to the chunked transformer hurts performance relative to the chunked baseline on several LRA tasks, while the ELD mask helps.
Limitations
- LaS-Attention is causal, and the authors note that global convolution baselines benefit from bidirectionality, so the gap to SOTA remains.
- Performance is worse on non-local, non-smooth synthetic tasks.
- On larger language modeling settings (Wikitext-103, 12 layers) LaS shows slight degradation.
Experimental setup
Datasets: Long Range Arena sequential MNIST permuted MNIST Text8 Enwik8 Wikitext-103 Atomic tasks (Select, CumSum, CumMax)
Benchmarks: Long Range Arena sequential MNIST permuted MNIST Text8 Enwik8 Atomic tasks
Metrics: accuracy bits per byte R2 score training speed peak memory
Baselines: Transformer Reformer Linformer BigBird Performer Luna-256 S4 SGConv MEGA LSTM
Builds on: Alibi S4 Mega SGConv Luna
Closest papers by meaning
- Transformer Layers as Painters (AAAI 2025 · similarity 0.89)
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture (ICML 2025 · similarity 0.89)
- AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers (ICML 2024 · similarity 0.89)
- What Does It Mean to Be a Transformer? Insights from a Theoretical Hessian Analysis (ICLR 2025 · similarity 0.88)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.88)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.