Research map / Architectures and efficiency / Attention and transformers
In-Context Algorithm Emulation in Fixed-Weight Transformers
Jerry Yao-Chieh Hu, Hude Liu, Jennifer Yuntong Zhang, Han Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase number of attention heads from 1 to 12 (ablation) | multi-head attention emulating a softmax head · Attention-head emulation | MSE: 3.469 -> 0.686 | Improved |
| use one frozen prompt-programmed attention instead of per-algorithm trained attention | frozen attention · Synthetic Lasso | MSE: 0.068 -> 0.059 | Improved |
| use one frozen prompt-programmed attention instead of per-algorithm trained attention | frozen attention · Synthetic linear regression | MSE: 0.147 -> 0.120 | Improved |
| use one frozen prompt-programmed attention instead of per-algorithm trained attention | frozen attention · Synthetic ridge regression | MSE: 0.004 -> 0.071 | Hurt |
| use one frozen prompt-programmed attention instead of per-algorithm trained attention | frozen attention · Ames Housing ridge regression | MSE: 0.0132 -> 0.0252 | Hurt |
| use one frozen prompt-programmed attention instead of per-algorithm trained attention | frozen attention · Ames Housing Lasso | MSE: 0.0354 -> 0.0322 | Improved |
Closest papers by meaning
- In-context Convergence of Transformers (ICML 2024 · similarity 0.92)
- Transformers are Universal In-context Learners (ICLR 2025 · similarity 0.89)
- From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers (NeurIPS 2025 · similarity 0.89)
- Optimality and NP-Hardness of Transformers in Learning Markovian Dynamical Functions (NeurIPS 2025 · similarity 0.89)
- Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions (ICLR 2024 · similarity 0.89)
- In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
