Research map / Architectures and efficiency / Attention and transformers
Monotonic Chunkwise Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace offline soft attention with MoChA, w = 2 | Online WSJ speech recognition · WSJ test | WER: 14.2% -> 13.9%; 14.6 ± 0.3% -> 15.0 ± 0.6% | Mixed |
| Replace hard monotonic attention with MoChA, w = 8 | Pointer-generator summarization model · CNN/Daily Mail | ROUGE-1: 31.14 -> 35.46 | Helped |
| Replace hard monotonic attention with MoChA, w = 8 | Pointer-generator summarization model · CNN/Daily Mail | ROUGE-2: 11.16 -> 13.55 | Helped |
| Replace soft attention with hard monotonic attention | Pointer-generator summarization model · CNN/Daily Mail | ROUGE-1: 39.11 -> 31.14 | Hurt |
| Replace soft attention with hard monotonic attention | Pointer-generator summarization model · CNN/Daily Mail | ROUGE-2: 15.76 -> 11.16 | Hurt |
| Double the monotonic attention energy hidden dimensionality (ablation) | Monotonic attention on WSJ speech recognition · WSJ speech recognition | WER: decrease of 0.3% WER | No clear effect |
| Halve the attention energy hidden dimensionality (ablation) | MoChA, w = 2, on WSJ speech recognition · WSJ speech recognition | WER: increasing the WER by only 0.2% | No clear effect |
| Increase the temporal receptive field of the convolutional front-end (ablation) | Monotonic attention on WSJ speech recognition · WSJ speech recognition | WER: an increase of 0.3% WER | No clear effect |
| Use MAtChA with adaptive chunk boundaries instead of fixed-size MoChA chunks (ablation) | Online attention models · Tasks tried in the paper | performance | No clear effect |
Closest papers by meaning
- Training-Free Long-Context Scaling of Large Language Models (ICML 2024 · similarity 0.86)
- Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models (NeurIPS 2026 · similarity 0.86)
- MVA: Linear Attention with High-order Query-Keys Integration and Multi-level Vocabulary Decomposition (ICML 2025 · similarity 0.85)
- Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study (ICLR 2025 · similarity 0.85)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.85)
- More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.