Research map / Architectures and efficiency / Attention and transformers
Structured Attention Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add syntactic structured attention over soft parents | tree-transduction encoder · Tree transduction, depth 2 | average length to failure (%): 87.4 -> 99.2 | Helped |
| add segmentation CRF attention instead of sigmoid attention | Japanese character-to-word translation · Japanese character-to-word translation | BLEU: 13.1 -> 14.6 | Helped |
| add segmentation CRF attention instead of simple attention | Japanese word-to-word translation · Japanese word-to-word translation | BLEU: 14.1 -> 14.3 | No clear effect |
| add binary pairwise potentials to the CRF (ablation) | bAbI QA · bAbI 1K average | answer accuracy (%): 73.8 -> 81.0 | Helped |
| add binary pairwise potentials to the CRF (ablation) | bAbI QA · bAbI 1K average | supporting fact selection accuracy (%): 17.4 -> 53.7 | Helped |
| use binary CRF attention instead of MemN2N attention | bAbI QA · bAbI 1K average | answer accuracy (%): 81.4 -> 81.0 | No clear effect |
| use binary CRF attention instead of MemN2N attention | bAbI QA · bAbI 1K average | supporting fact selection accuracy (%): 39.6 -> 53.7 | Helped |
| add syntactic structured attention | Stanford NLI model · Stanford NLI test | accuracy (%): 86.2 -> 86.8 | Helped |
| pretrain the syntactic attention layer on parse trees (ablation) | Stanford NLI model · Stanford NLI test | accuracy (%): 86.8 -> 86.5 | Hurt |
| add simple intra-sentence attention | Stanford NLI model · Stanford NLI test | accuracy (%): 85.8 -> 86.2 | Helped |
| add pipeline hard-parent features | Stanford NLI model · Stanford NLI test | accuracy (%): 85.8 -> 86.1 | Helped |
| use structured rather than simple self-attention | tree-transduction encoder · Tree transduction, depth 3 | average length to failure (%): 49.6 -> 87.0 | Helped |
Closest papers by meaning
- A Regularized Framework for Sparse and Structured Neural Attention (NeurIPS 2017 · similarity 0.89)
- Customizing the Inductive Biases of Softmax Attention using Structured Matrices (ICML 2025 · similarity 0.87)
- Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data (ICML 2025 · similarity 0.87)
- On Understanding Attention-Based In-Context Learning for Categorical Data (ICML 2025 · similarity 0.86)
- Fundamental limits of learning in sequence multi-index models and deep attention networks: high-dimensional asymptotics and sharp thresholds (ICML 2025 · similarity 0.86)
- PANDA: Prior-guided Attentional Dual-path Architecture (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.