Research map / Architectures and efficiency / Attention and transformers
An evolutionary perspective on modes of learning in Transformers
Alexander Ku, Thomas L. Griffiths, Stephanie C.Y. Chan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase environmental stability (alpha toward 1) in sinusoid regression (ablation) | Transformer · Sinusoid regression | S_ICL | Hurt |
| increase mapping persistence (alpha toward 1) in Omniglot (ablation) | Transformer · Omniglot classification | S_ICL | Hurt |
| increase cue reliability (lower noise variance) in volatile sinusoid environments (ablation) | Transformer · Sinusoid regression | S_ICL | Improved |
| increase label correctness probability rho when alpha < 0.95 (ablation) | Transformer · Omniglot classification | S_ICL | Improved |
| perfectly reliable cues (rho = 1) in a perfectly stable environment (ablation) | Transformer · Omniglot classification | S_ICL | No clear effect |
| reduce Omniglot vocabulary size from 1623 to 100 (ablation) | Transformer · Omniglot classification (alpha=0.999, rho=0.95) | early-training S_ICL: 1623 -> 100 classes; S_ICL < 0.2 | Hurt |
| compare prequential codelength of ICL vs IWL on Omniglot | Transformer · Omniglot classification | prequential codelength | Improved |
| compare prequential codelength of IWL vs ICL on sinusoid | Transformer · Sinusoid regression | prequential codelength | Improved |
Closest papers by meaning
- What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formation (ICML 2024 · similarity 0.87)
- Breaking through the learning plateaus of in-context learning in Transformer (ICML 2024 · similarity 0.86)
- Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.85)
- From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers (NeurIPS 2025 · similarity 0.85)
- Small-scale proxies for large-scale Transformer training instabilities (ICLR 2024 · similarity 0.85)
- Differential learning kinetics govern the transition from memorization to generalization during in-context learning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
