Research map / Architectures and efficiency / Attention and transformers
Understanding and Relaxing the Limitations of Transformers for Linear Algebra
Andres Potapczynski, Alex Ali, Andrew Gordon Wilson
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on diverse matrix sizes instead of fixed size N=50 (ablation) | NumFormer · Gaussian symmetric matrices, Tr(A) | relative error: N = 50 to N = 45 worsens the performance almost 10 times | Hurt |
| add looping (tied weights) to the transformer (ablation) | NumFormer · Toeplitz A^-1 | error across layers | Improved |
| add range embedding (learned projection, no flattening) (ablation) | NumFormer · matrix market A^-1 20x20 | test relative error | Improved |
| train on structured matrices and varied eigenspectra instead of Gaussian only (ablation) | RangeFormer · OOD matrices (Id, zeros, Toeplitz, matrix market) | relative error | Improved |
| add curriculum from N=100 checkpoint (ablation) | RangeFormer · linear solves N=1K | training loss convergence | Improved |
| increase embedding size D beyond 128 (ablation) | RangeFormer · embedding size scaling | error: D=128: 1.9 × 10−1; D=1024: 1.2 × 10−1 at N=20 | No clear effect |
| replace Cholesky with RangeFormer for GP kernel learning | GP pipeline · synthetic sin GP regression | test RMSE: 0.87996 -> 0.87978 | No clear effect |
Closest papers by meaning
- Linear Transformers Implicitly Discover Unified Numerical Algorithms (NeurIPS 2025 · similarity 0.86)
- Graph Transformers Dream of Electric Flow (ICLR 2025 · similarity 0.86)
- CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs (NeurIPS 2026 · similarity 0.86)
- Predictive Geometry of Hidden Trajectories in Transformers (NeurIPS 2026 · similarity 0.85)
- Residual Matrix Transformers: Scaling the Size of the Residual Stream (ICML 2025 · similarity 0.84)
- Linear Transformers are Versatile In-Context Learners (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
