Research map / Architectures and efficiency / Attention and transformers
Algorithmic Capabilities of Random Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that randomly initialized transformers with only embedding and unembedding layers trained can perform algorithmic tasks (modular addition, associative recall, decimal addition, parenthesis balancing) and generate grammatical text, explained by low-dimensional subspace selection.
Problem
Whether algorithmic capabilities of transformers originate from training or are already present in randomly initialized models, and how much the supervisory signal versus initialization determines them.
Approach
Freeze all intermediate transformer layers at random initialization and optimize only the input embedding and output unembedding matrices; show that this suffices for many algorithmic tasks, and explain the success by steering hidden computation into low-dimensional subspaces ("subspace selection") rather than sparse sub-networks.
Key results
- Random transformers (width 1024, only embeddings and unembeddings trained) reach 100% test accuracy on modular addition, needle-in-a-haystack, decimal addition, and parenthesis balancing, while a fully trained LSTM reaches 53.0% on decimal addition.
- Unembedding-only training reaches just 0.4% test accuracy on modular addition, and embedding-only with fixed unembedding reaches 100% on needle-in-a-haystack but fails on other variants; all three matrices must be trained for reliable results.
- Random transformers memorize 4.53% of memorization examples (0.41 bits per trainable parameter) versus 80.54% (2.88 bits) for fully trained models; random 512-width model reaches 2.64 cross-entropy on TinyStories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add trainable embeddings and unembedding instead of training only U (ablation) | Random transformer · Modular Addition test | accuracy: 0.4% -> 100.0% | Helped |
| Add a trainable unembedding instead of training only E (ablation) | Random transformer · Modular Addition test | accuracy: 1.2% -> 100.0% | Helped |
| Train positional embeddings in addition to E_token and U (ablation) | Random transformer · Modular Addition test | accuracy: 100.0% -> 100.0% | No clear effect |
| Add trainable embeddings and unembedding instead of training only U (ablation) | Random transformer · Needle-in-a-Haystack | accuracy: 17.7% -> 98.5% | Helped |
| Add a trainable unembedding instead of training only E (ablation) | Random transformer · Needle-in-a-Haystack | accuracy: 100.0% -> 98.5% | No clear effect |
| Train positional embeddings in addition to E_token and U (ablation) | Random transformer · Needle-in-a-Haystack | accuracy: 98.5% -> 100.0% | No clear effect |
| Add trainable embeddings and unembedding instead of training only U (ablation) | Random transformer · Decimal Addition | accuracy: 27.3% -> 48.5% | Helped |
| Add a trainable unembedding instead of training only E (ablation) | Random transformer · Decimal Addition | accuracy: 100.0% -> 48.5% | Hurt |
| Train positional embeddings in addition to E_token and U (ablation) | Random transformer · Decimal Addition | accuracy: 48.5% -> 100.0% | Helped |
| Add trainable embeddings and unembedding instead of training only U (ablation) | Random transformer · Parenthesis Balancing | accuracy: 92.3% -> 100.0% | Helped |
| Add a trainable unembedding instead of training only E (ablation) | Random transformer · Parenthesis Balancing | accuracy: 100.0% -> 100.0% | No clear effect |
| Train positional embeddings in addition to E_token and U (ablation) | Random transformer · Parenthesis Balancing | accuracy: 100.0% -> 100.0% | No clear effect |
Surprising findings
- Random transformers with only embeddings trained outperform fully trained LSTMs on several algorithmic tasks.
- Random transformers on TinyStories produce largely grammatical, topically coherent text despite worse perplexity than fully trained models.
Limitations
- Only standard decoder-only transformers examined; linear attention and state-space models not tested.
- Results limited to a small part of the architecture design space (size, attention, normalization, tokenization).
- Analysis restricted to linear subspaces via PCA; nonlinear latent structure not addressed.
Experimental setup
Datasets: TinyStories synthetic modular addition synthetic needle-in-a-haystack synthetic decimal addition synthetic parenthesis balancing synthetic memorization
Benchmarks: modular addition needle-in-a-haystack decimal addition parenthesis balancing memorization TinyStories language modeling circuit imitation
Metrics: test accuracy cross-entropy loss explained variance of top principal components KL divergence bits per parameter
Baselines: fully trained transformer fully trained LSTM linearized transformer unembedding-only training embedding-only variants
Builds on: GPT-2 lottery ticket hypothesis reservoir computing grokking
Closest papers by meaning
- From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers (NeurIPS 2025 · similarity 0.89)
- Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions (ICLR 2024 · similarity 0.88)
- Do Efficient Transformers Really Save Computation? (ICML 2024 · similarity 0.88)
- Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks (ICML 2024 · similarity 0.88)
- Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges (ICML 2025 · similarity 0.88)
- Learning Randomized Algorithms with Transformers (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.