Research map / Architectures and efficiency / Attention and transformers
A Tensorized Transformer for Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace multi-head attention with Multi-linear attention using two core tensors | Transformer · One-Billion | test PPL: 21.8 -> 19.5 | Helped |
| Replace multi-head attention with Multi-linear attention using two core tensors | Transformer · PTB | test PPL: 54.52 -> 49.8 | Helped |
| Replace multi-head attention with Multi-linear attention using two core tensors | Transformer · WikiText-103 | test PPL: 18.3 -> 18.9 | Mixed |
| Replace multi-head attention with Multi-linear attention using two core tensors | Transformer · WMT-2016 English-German | BLEU: 34.5* -> 34.91 | Helped |
| Add a second core tensor instead of using one (ablation) | Tensorized Transformer · One-Billion | test PPL: 20.5 -> 19.5 | Helped |
| Add a second core tensor instead of using one (ablation) | Tensorized Transformer · PTB | test PPL: 57.9 -> 49.8 | Helped |
| Add a second core tensor instead of using one (ablation) | Tensorized Transformer · WikiText-103 | test PPL: 20.9 -> 18.9 | Helped |
| Add a second core tensor instead of using one (ablation) | Tensorized Transformer · WMT-2016 English-German | BLEU: 34.10 -> 34.91 | Helped |
| Increase the number of core tensors beyond two (ablation) | Tensorized Transformer · Language modeling | PPL | Hurt |
| Use Multi-linear attention instead of Transformer-XL attention at matched parameter sizes | Transformer-XL · PTB and WikiText-103 | test PPL: 87.8 -> 57.9; 34.8 -> 20.9 | Helped |
Closest papers by meaning
- Tensor Product Attention Is All You Need (NeurIPS 2025 · similarity 0.89)
- Tensorized Attention for Understanding Multi-Object Relationships (AAAI 2025 · similarity 0.87)
- Efficient Large Language Model Inference with Neural Block Linearization (NeurIPS 2025 · similarity 0.86)
- PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels (ICML 2024 · similarity 0.86)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.86)
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.