AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks

Damien Teney, Liangze Jiang, Hemanth Saratchandran, Simon Lucey

ICLR 2026 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace GeLU in MLP with learned splineTransformer (softmax attention) · TinyStoriesvalidation accuracy (%): 63.7 -> 64.4Improved
replace Swish in GLU with learned splineGLU MLP · TinyStoriesvalidation accuracy (%): 63.7 -> 64.0Improved
replace softmax attention with learned componentsTransformer · TinyStoriesperformanceNo clear effect
replace GeLU with Sinc, TanH, or Linear activationTransformer · TinyStoriesvalidation accuracy (%): 63.7 -> 47.7Hurt
use polynomial attention P1 instead of softmaxTransformer · TinyStoriesvalidation accuracy (%): 63.7 -> 63.0Hurt
use Adaptive softmax or NormSoftmax instead of softmaxTransformer · TinyStoriesvalidation accuracy (%): 63.7 -> 63.7No clear effect
increase parallel models in stage I from M=1 to M=6 (ablation)Optimized MLP · TinyStories (2 layers)validation accuracy (%): 63.8 -> 64.3Improved
increase parallel models in stage I from M=1 to M=6 (ablation)Optimized MLP · TinyStories (4 layers)validation accuracy (%): 65.3 -> 65.4Improved
initialize learned non-linearity from GeLU instead of zero (ablation)Optimized MLP · language modeling datasetsperplexityHurt
make learned non-linearities more periodic or symmetric (ablation)Optimized MLP · TinyStoriestest accuracy (%): (-0.2)Hurt
replace ReLU with learned splineNanoGPT Speedrun baseline · FineWeb (4 layers)validation loss: 3.87 -> 3.82Improved
replace ReLU with learned splineNanoGPT Speedrun baseline · FineWeb (12 layers)validation loss: 3.72 -> 3.68Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy