Research map / Architectures and efficiency / Attention and transformers
Frozen Pretrained Transformers as Universal Computation Engines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use natural-language-pretrained frozen layers rather than random initialization | FPT · ListOps | accuracy: 34.3% -> 38.4% | Helped |
| Use natural-language-pretrained frozen layers rather than random initialization | FPT · MNIST | accuracy: 91.7% -> 98.0% | Helped |
| Use natural-language-pretrained frozen layers rather than random initialization | FPT · CIFAR-10 | accuracy: 61.7% -> 68.2% | Helped |
| Use natural-language-pretrained frozen layers rather than random initialization | FPT · CIFAR-10 LRA | accuracy: 36.1% -> 38.6% | Helped |
| Use natural-language-pretrained frozen layers rather than random initialization | FPT · Remote Homology | accuracy: 9.3% -> 12.7% | Helped |
| Pretrain on Bit Memory rather than natural language | Frozen transformer · Remote Homology | accuracy: 12.7% -> 7.8% | Hurt |
| Pretrain on images rather than natural language | Frozen transformer · CIFAR-10 | accuracy: 68.2% -> 72.5% | Helped |
| Pretrain on images rather than natural language | Frozen transformer · Remote Homology | accuracy: 12.7% -> 7.5% | Hurt |
| Also finetune feedforward layers (ablation) | FPT · CIFAR-10 | accuracy: 68.2% -> 76.6% | Helped |
| Also finetune attention layers (ablation) | FPT · CIFAR-10 | accuracy: 68.2% -> 47.7% | Hurt |
| Also finetune the last attention layer (ablation) | FPT · CIFAR-10 | accuracy: 68.2% -> 80.0% | Helped |
| Add layer norm finetuning to output-only finetuning (ablation) | Pretrained frozen transformer · MNIST | accuracy: 23% -> 96% | Helped |
Closest papers by meaning
- Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models (NeurIPS 2025 · similarity 0.88)
- Trainable Transformer in Transformer (ICML 2024 · similarity 0.88)
- How Do Nonlinear Transformers Learn and Generalize in In-Context Learning? (ICML 2024 · similarity 0.87)
- Algorithmic Capabilities of Random Transformers (NeurIPS 2024 · similarity 0.87)
- On the Generalization Ability of Next-Token-Prediction Pretraining (ICML 2025 · similarity 0.87)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.