Research map / Architectures and efficiency / Attention and transformers
Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems
Subhabrata Dutta, Tanya Gautam, Soumen Chakrabarti, Tanmoy Chakraborty
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Transformer with TransEvolve-fullFF-2 | Transformer BASE · WMT 2014 En-Fr | BLEU: 38.1 -> 39.5 | Improved |
| replace Transformer with TransEvolve-fullFF-2 | Transformer BASE · WMT 2013 En-De | BLEU: 25.8 -> 26.2 | Improved |
| replace Transformer with TransEvolve-fullFF-2 | Transformer BASE · WMT 2014 En-De | BLEU: 27.3 -> 27.2 | No clear effect |
| replace feed-forward layers with random rotation matrices (randomFF vs fullFF) (ablation) | TransEvolve-1 · WMT 2014 En-Fr | BLEU: 38.0 -> 32.6 | Hurt |
| replace feed-forward layers with random rotation matrices (randomFF vs fullFF) (ablation) | TransEvolve-2 · WMT 2014 En-De | BLEU: 27.2 -> 23.8 | Hurt |
| use 2 blocks of 3-layer encoders instead of one 6-layer block (ablation) | TransEvolve-fullFF · WMT 2014 En-Fr | BLEU: 38.0 -> 39.5 | Improved |
| use randomFF instead of fullFF (ablation) | TransEvolve-1 · ListOps | accuracy (%): 42.2 -> 43.2 | Improved |
| use 2 blocks of 3-layer encoders instead of one 6-layer block (ablation) | TransEvolve-randomFF · ListOps | accuracy (%): 43.2 -> 39.1 | Hurt |
| use randomFF instead of fullFF (ablation) | TransEvolve-2 · char-level IMDB | accuracy (%): 65.6 -> 66.1 | Improved |
| use randomFF instead of fullFF (ablation) | TransEvolve-1 · AGnews | accuracy (%): 91.1 -> 90.6 | Hurt |
| use randomFF instead of fullFF (ablation) | TransEvolve-2 · IMDB | accuracy (%): 87.6 -> 87.5 | No clear effect |
| replace Transformer with TransEvolve-fullFF-1 | Transformer · AGnews | accuracy (%): 88.8 -> 91.1 | Improved |
Closest papers by meaning
- PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations (ICML 2025 · similarity 0.88)
- SEA: State-Exchange Attention for High-Fidelity Physics Based Transformers (NeurIPS 2024 · similarity 0.87)
- Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators (NeurIPS 2024 · similarity 0.86)
- Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems (ICML 2025 · similarity 0.86)
- Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime (NeurIPS 2026 · similarity 0.86)
- A multiscale analysis of mean-field transformers in the moderate interaction regime (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
