Research map / Architectures and efficiency / Attention and transformers
Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replaces Transformer attention blocks on IWSLT2017 with shallow one-hidden-layer feed-forward networks trained via knowledge distillation from a vanilla Transformer, finding self-attention replaceable but cross-attention not.
Problem
Whether standard shallow feed-forward networks can mimic the behavior of Transformer attention modules in sequence-to-sequence translation, and how much performance is retained.
Approach
Substitute encoder and decoder attention sub-blocks (ALR, ALRR, ASLR, ELR variants) with shallow one-hidden-layer feed-forward networks trained by distilling intermediate activations of a teacher Transformer, with sentences padded to length 50 and masked.
Key results
- ALR encoder self-attention replacement at size L reaches BLEU 0.252 (E2G) vs baseline 0.257, close to the original Transformer.
- Decoder cross-attention replacement performs much worse, e.g., ALR Dec CA XS reaches 0.035 BLEU on E2G vs 0.257 baseline.
- Replacement networks carry more parameters than the original 60,000-parameter attention layer (e.g., 320K to 41M per network).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace encoder self-attention with ALR feed-forward networks | Transformer · IWSLT2017, four translation directions | BLEU: 0.257 -> 0.252; 0.324 -> 0.327; 0.276 -> 0.276; 0.292 -> 0.288 | No clear effect |
| Replace decoder self-attention with ALR feed-forward networks | Transformer · IWSLT2017, four translation directions | BLEU: 0.257 -> 0.253; 0.324 -> 0.323; 0.276 -> 0.273; 0.292 -> 0.291 | No clear effect |
| Replace decoder cross-attention with ALR feed-forward networks (ablation) | Transformer · IWSLT2017, four translation directions | BLEU: 0.257 -> 0.115; 0.324 -> 0.130; 0.276 -> 0.109; 0.292 -> 0.115 | Hurt |
| Replace all attention modules, including decoder cross-attention, with ALR feed-forward networks (ablation) | Transformer · IWSLT2017, four translation directions | BLEU: 0.257 -> 0.105; 0.324 -> 0.134; 0.276 -> 0.117; 0.292 -> 0.116 | Hurt |
| Replace attention plus the residual connection (ALRR), rather than attention alone (ALR) (ablation) | Transformer encoder self-attention · IWSLT2017, four translation directions | BLEU: ALR L vs ALRR L: 0.252 vs 0.243; 0.327 vs 0.315; 0.276 vs 0.263; 0.288 vs 0.276 | Hurt |
| Replace each attention head with a separate FF network (ASLR), rather than one network for the attention layer (ALR) (ablation) | Transformer encoder self-attention · IWSLT2017, four translation directions | BLEU: ALR L vs ASLR L: 0.252 vs 0.252; 0.327 vs 0.326; 0.276 vs 0.271; 0.288 vs 0.290 | Mixed |
| Replace the entire encoder layer (ELR), rather than only attention (ALR) (ablation) | Transformer encoder · IWSLT2017, four translation directions | BLEU: ALR L vs ELR L: 0.252 vs 0.194; 0.327 vs 0.248; 0.276 vs 0.225; 0.288 vs 0.219 | Hurt |
| Increase ALR encoder FF network size from XS to L (ablation) | Transformer encoder self-attention · IWSLT2017, four translation directions | BLEU: 0.180 -> 0.252; 0.235 -> 0.327; 0.226 -> 0.276; 0.218 -> 0.288 | Helped |
| Increase ALR decoder cross-attention FF network size from XS to L (ablation) | Transformer decoder cross-attention · IWSLT2017, four translation directions | BLEU: 0.035 -> 0.115; 0.036 -> 0.130; 0.035 -> 0.109; 0.039 -> 0.115 | Helped |
| Scale replacement networks beyond the reported sizes (ablation) | Proposed FF replacement networks · IWSLT2017 | BLEU | No clear effect |
Surprising findings
- Replacing decoder cross-attention with feed-forward networks substantially degrades translation, while self-attention replacement is largely successful.
- The authors note that attentionless Transformers could not be trained from scratch and needed knowledge distillation to converge.
Limitations
- Replacement networks have more parameters than the attention layers they replace.
- Fixed maximum sentence length of 50 tokens limits flexibility for variable-length inputs.
- Cross-attention cannot be replaced effectively by shallow FF networks.
Experimental setup
Datasets: IWSLT2017
Benchmarks: IWSLT2017 E2G IWSLT2017 G2E IWSLT2017 E2F IWSLT2017 F2E
Metrics: BLEU relative BLEU vs baseline
Baselines: Transformer (vanilla, embedding size 128)
Builds on: Transformer knowledge distillation
Closest papers by meaning
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.90)
- An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.89)
- What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks (ICML 2024 · similarity 0.89)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.89)
- The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training (ICML 2025 · similarity 0.89)
- Memorization Capacity of Multi-Head Attention in Transformers (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.