Research map / Architectures and efficiency / Attention and transformers
Area Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace item-based attention with parameter-free area attention | Transformer Big · WMT14 EN-DE, token-level | BLEU: 29.26 -> 29.77 | Helped |
| Replace item-based attention with parameter-free area attention | Transformer Big · WMT14 EN-FR, token-level | BLEU: 41.0 -> 41.46 | Helped |
| Use area attention | LSTM 1024 cells, 4 heads · WMT14 EN-DE, token-level | BLEU: 20.21 -> 24.85 | Helped |
| Use area attention | LSTM 1024 cells, 4 heads · WMT14 EN-FR, token-level | BLEU: 32.21 -> 35.97 | Helped |
| Replace item-based attention with area attention | Transformer Big · WMT14 EN-DE, character-level | BLEU: 25.96 -> 26.57 | Helped |
| Replace item-based attention with area attention | Transformer Big · WMT14 EN-FR, character-level | BLEU: 33.82 -> 35.16 | Helped |
| Add area attention with maximum size 3×3 | Image captioning Transformer · Flickr 1K | CIDEr: 0.355 -> 0.377 | Helped |
| Add area attention with maximum size 2×2 | Image captioning Transformer · Flickr 1K | ROUGE-L: 0.409 -> 0.420 | Helped |
| Increase maximum area size from 2×2 to 3×3 (ablation) | Image captioning Transformer · Flickr 1K | CIDEr: 0.364 -> 0.377 | Helped |
| Increase maximum area size from 2×2 to 3×3 (ablation) | Image captioning Transformer · COCO40 | CIDEr: 1.060 -> 1.060 | No clear effect |
| Add trainable enriched area features instead of the parameter-free area representation (ablation) | Image captioning Transformer, 3×3 area attention · COCO40 | CIDEr: 1.060 -> 1.045 | Hurt |
| Add trainable enriched area features instead of the parameter-free area representation (ablation) | Image captioning Transformer, 3×3 area attention · COCO40 | ROUGE-L: 0.706 -> 0.707 | Helped |
Closest papers by meaning
- Benchmarking Attention for Tabular Foundation Models (NeurIPS 2026 · similarity 0.86)
- Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.85)
- Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study (ICLR 2025 · similarity 0.85)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.85)
- Dual Dimensionality for Local and Global Attention (NeurIPS 2026 · similarity 0.85)
- A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.