Research map / Architectures and efficiency / Attention and transformers
Towards Understanding Inductive Bias in Transformers: A View From Infinity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes transformer inductive bias in the infinite-width GP (NNGP) limit using symmetric-group representation theory, giving learnability scaling laws in context length for permutation-symmetric data, and shows WikiText-2 has approximate permutation symmetry in first-order correlations.
Problem
Characterizing the inductive bias of transformers, i.e., which functions are expressible and how many samples are needed to learn them, as a function of context length, especially for sequence data with permutation symmetry.
Approach
Treat the infinite-width transformer as a GP with NNGP kernel; use the equivalent kernel eigenproblem and representation theory of the symmetric group (irreps labeled by partitions (L-k,k)) to bound eigenvalues by irrep dimension, giving learnability scaling N ~ L^k for irrep (L-k,k).
Key results
- Predicted GP-limit MSE matches exact GP inference in-distribution and out-of-distribution for linear-activation transformer block.
- Softmax attention kernel eigenvalues scale as L^0 for trivial irrep and L^-1 for standard irrep, matching the irrep-dimension bounds.
- WikiText-2 first-order correlation matrices for nonzero cyclic frequencies k are nearly identical in cosine similarity and spectrum, while k=0 differs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use softmax attention instead of fully connected layers | Transformer-like model on a symmetric task · Symmetric task | samples required to learn target: N=\Theta(1) vs. N=O(L) | Helped |
| Increase context length L for targets in irrep (L-k,k) | Transformer-like model with permutation symmetry · Symmetric sequence tasks | samples required to learn: Ω(L^{k}) | Hurt |
| Use a nonlinear erf MLP instead of a linear MLP (ablation) | Linear-attention transformer, GP limit · HMM mixture | learnability | No clear effect |
| Use a nonlinear erf MLP instead of a linear MLP (ablation) | Linear-attention transformer, GP limit · HMM mixture | predictor similarity | No clear effect |
| Train a finite erf-MLP network with SGD instead of using exact GP inference (ablation) | Linear-attention transformer · HMM mixture, OOD | prediction agreement | No clear effect |
| Use standard softmax attention instead of linear attention (ablation) | Transformer block · Symmetric task | eigenvalue scaling in the trivial irrep: L^0 | Helped |
| Use standard softmax attention instead of linear attention (ablation) | Transformer block · Symmetric task | eigenvalue scaling in the standard irrep: L^{-1} | Helped |
| Evaluate on the OOD distribution instead of the training distribution | Linear-attention transformer, GP limit · HMM mixture | generalization loss | Hurt |
Limitations
- Analysis is in the infinite-width GP limit and only partially extends to finite networks.
- Main example uses linearized attention and a toy two-state HMM dataset.
- Permutation symmetry in WikiText is shown only at first-order (linear) correlations.
Experimental setup
Datasets: mixture of hidden Markov models WikiText-2
Benchmarks: WikiText-2
Metrics: MSE loss learnability cosine similarity of correlation matrices eigenvalue scaling
Baselines: fully connected network exact GP inference
Builds on: NNGP equivalent kernel Langevin dynamics Specht modules
Closest papers by meaning
- From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers (NeurIPS 2025 · similarity 0.87)
- Permutation Equivariance of Transformers and Its Applications (CVPR 2024 · similarity 0.86)
- How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpad (NeurIPS 2024 · similarity 0.86)
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.86)
- Algorithmic Capabilities of Random Transformers (NeurIPS 2024 · similarity 0.85)
- (How) Can Transformers Predict Pseudo-Random Numbers? (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.