Research map / Architectures and efficiency / RNNs and normalization
Understanding the Parameter Space Geometry of Transformers Encoding Boolean Functions
Blanka Kövér, Alexandra Butoi, Anej Svete, Michael Hahn, Ryan Cotterell
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train transformer on Majority instead of Parity | transformer encoder · sensitivity profile after training | match to target sensitivity profile | Improved |
| train on Parity (maximally sensitive function) | transformer encoder · Parity, trained | sensitivity profile | Hurt |
| vary initialization scheme (uniform, Gaussian, Xavier uniform, Xavier Gaussian) (ablation) | randomly initialized transformer · random-init sensitivity spectra | skew of sensitivity spectrum toward zero | No clear effect |
Closest papers by meaning
- Towards Understanding Inductive Bias in Transformers: A View From Infinity (ICML 2024 · similarity 0.87)
- Predictive Geometry of Hidden Trajectories in Transformers (NeurIPS 2026 · similarity 0.86)
- Transformers Learn Low Sensitivity Functions: Investigations and Implications (ICLR 2025 · similarity 0.86)
- Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers (NeurIPS 2025 · similarity 0.86)
- How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpad (NeurIPS 2024 · similarity 0.85)
- Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
