Research map / Architectures and efficiency / Attention and transformers
Context-Aware Self-Attention Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Context-Aware SANs | Transformer-Base · WMT17 Zh⇒En newstest2017 | BLEU: 24.13 -> 24.67 | Helped |
| Add Context-Aware SANs | Transformer-Big · WMT17 Zh⇒En newstest2017 | BLEU: 24.56 -> 25.15 | Helped |
| Add Context-Aware SANs | Transformer-Base · WMT14 En⇒De newstest2014 | BLEU: 27.31 -> 28.26 | Helped |
| Add Context-Aware SANs | Transformer-Big · WMT14 En⇒De newstest2014 | BLEU: 28.58 -> 28.89 | Helped |
| Add global context to encoder self-attention (ablation) | Transformer-Base · WMT14 En⇒De newstest2014 | BLEU: 27.31 -> 27.96 | Helped |
| Add deep-global context to encoder self-attention (ablation) | Transformer-Base · WMT14 En⇒De newstest2014 | BLEU: 27.31 -> 28.15 | Helped |
| Add deep context to encoder self-attention (ablation) | Transformer-Base · WMT14 En⇒De newstest2014 | BLEU: 27.31 -> 28.01 | Helped |
| Combine deep-global and deep context in encoder self-attention (ablation) | Transformer-Base · WMT14 En⇒De newstest2014 | BLEU: 27.31 -> 28.26 | Helped |
| Add deep-global context to decoder self-attention (ablation) | Transformer-Base · WMT14 En⇒De newstest2014 | BLEU: 27.31 -> 27.94 | Helped |
| Combine deep-global and deep context in decoder self-attention (ablation) | Transformer-Base · WMT14 En⇒De newstest2014 | BLEU: 27.31 -> 28.02 | Helped |
| Apply context-aware self-attention to both encoder and decoder rather than the encoder-side configuration (ablation) | Transformer-Base · WMT14 En⇒De newstest2014 | BLEU: 28.26 -> 28.16 | Hurt |
| Add context to both queries and keys (ablation) | Transformer-Base · WMT14 En⇒De newstest2013 | BLEU: 25.84 -> 26.42 | Helped |
Closest papers by meaning
- Selective Attention: Enhancing Transformer through Principled Context Control (NeurIPS 2024 · similarity 0.88)
- Core Context Aware Transformers for Long Context Language Modeling (ICML 2025 · similarity 0.88)
- Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding (ICML 2025 · similarity 0.87)
- Where does In-context Learning Happen in Large Language Models? (NeurIPS 2024 · similarity 0.87)
- The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation (NeurIPS 2025 · similarity 0.86)
- Overcoming Long Context Limitations of State Space Models via Context Dependent Sparse Attention (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.