Research map / Optimization / Implicit bias and generalization
Transformers Trained via Gradient Descent Can Provably Learn a Class of Teacher Models
Chenyang Zhang, Qingyue Zhao, Quanquan Gu, Yuan Cao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train one-layer position-only transformer on teacher outputs (synthetic CNN/GCN) | One-layer transformer · Synthetic teacher-student | excess training loss slope: slope -1 | Improved |
| evaluate OOD loss of trained transformer | One-layer transformer · Synthetic teacher-student | excess OOD test loss slope: slope -0.5 | Improved |
| use real MNIST-pretrained CNN teacher instead of synthetic | One-layer transformer · MNIST teacher CNN | cosine similarity of W_V: above 0.9 | Improved |
Closest papers by meaning
- In-context Convergence of Transformers (ICML 2024 · similarity 0.89)
- In-context Learning on Function Classes Unveiled for Transformers (ICML 2024 · similarity 0.88)
- Transformers are almost optimal metalearners for linear classification (NeurIPS 2025 · similarity 0.88)
- Linear Transformers are Versatile In-Context Learners (NeurIPS 2024 · similarity 0.88)
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.88)
- Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
