Research map / Architectures and efficiency / Attention and transformers
Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
Damien Teney, Liangze Jiang, Hemanth Saratchandran, Simon Lucey
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GeLU in MLP with learned spline | Transformer (softmax attention) · TinyStories | validation accuracy (%): 63.7 -> 64.4 | Improved |
| replace Swish in GLU with learned spline | GLU MLP · TinyStories | validation accuracy (%): 63.7 -> 64.0 | Improved |
| replace softmax attention with learned components | Transformer · TinyStories | performance | No clear effect |
| replace GeLU with Sinc, TanH, or Linear activation | Transformer · TinyStories | validation accuracy (%): 63.7 -> 47.7 | Hurt |
| use polynomial attention P1 instead of softmax | Transformer · TinyStories | validation accuracy (%): 63.7 -> 63.0 | Hurt |
| use Adaptive softmax or NormSoftmax instead of softmax | Transformer · TinyStories | validation accuracy (%): 63.7 -> 63.7 | No clear effect |
| increase parallel models in stage I from M=1 to M=6 (ablation) | Optimized MLP · TinyStories (2 layers) | validation accuracy (%): 63.8 -> 64.3 | Improved |
| increase parallel models in stage I from M=1 to M=6 (ablation) | Optimized MLP · TinyStories (4 layers) | validation accuracy (%): 65.3 -> 65.4 | Improved |
| initialize learned non-linearity from GeLU instead of zero (ablation) | Optimized MLP · language modeling datasets | perplexity | Hurt |
| make learned non-linearities more periodic or symmetric (ablation) | Optimized MLP · TinyStories | test accuracy (%): (-0.2) | Hurt |
| replace ReLU with learned spline | NanoGPT Speedrun baseline · FineWeb (4 layers) | validation loss: 3.87 -> 3.82 | Improved |
| replace ReLU with learned spline | NanoGPT Speedrun baseline · FineWeb (12 layers) | validation loss: 3.72 -> 3.68 | Improved |
Closest papers by meaning
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.89)
- From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers (NeurIPS 2025 · similarity 0.89)
- Do Efficient Transformers Really Save Computation? (ICML 2024 · similarity 0.89)
- Understanding the Generalization of In-Context Learning in Transformers: An Empirical Study (ICLR 2025 · similarity 0.89)
- Algorithmic Capabilities of Random Transformers (NeurIPS 2024 · similarity 0.88)
- Transformers are almost optimal metalearners for linear classification (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
