Research map / Architectures and efficiency / Attention and transformers
Learning Pseudorandom Numbers with Transformers: Permuted Congruential Generators, Curricula, and Interpretability
Tao Tao, Maissam Barkeshli
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize from a smaller-modulus model (pretrained init) (ablation) | Transformer on XSLRR-20/10 · XSLRR-20/10 (m=2^20) | test accuracy at 640th token: 4% (random init) -> 0.9697 (pretrained+curriculum) | Improved |
| add curriculum mixing of smaller-modulus data (ablation) | Transformer on XSLRR large moduli · XSLRR-18/9 and 20/10 | test accuracy / stability | Improved |
| mix 1% small-modulus data via curriculum instead of fixed alpha (ablation) | XSLRR-18/9 · XSLRR-18/9 | accuracy at 256th token | Improved |
| exponential decay of alpha rather than cosine, linear or step (ablation) | curriculum schedule · XSLRR-18/9 | accuracy | Improved |
| add curriculum (ablation) | training at m=2^18 · XSLRR-18/9 | stable learning rate range | Improved |
| mix XSLRR-18/9 data instead of XSLRR-16/8 in curriculum for m=2^20 (ablation) | curriculum · XSLRR-20/10 | accuracy | No clear effect |
| increase model depth/size (ablation) | XSLRR-16/8 Transformer · XSLRR-16/8 | test loss/accuracy: 6-layer reaches 22.8% at 64th token | Improved |
| increase number of (a,c) pairs beyond 1024x1024 (ablation) | XSLRR-16/8 · XSLRR-16/8 | test loss | No clear effect |
| truncate output to fewer bits (k=1) (ablation) | truncated LCG · TLCG m=2^16 | accuracy: 95% at 256th element, lower at 64th | Hurt |
Closest papers by meaning
- (How) Can Transformers Predict Pseudo-Random Numbers? (ICML 2025 · similarity 0.94)
- Algorithmic Capabilities of Random Transformers (NeurIPS 2024 · similarity 0.87)
- Towards Understanding Inductive Bias in Transformers: A View From Infinity (ICML 2024 · similarity 0.86)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.85)
- Permutation Equivariance of Transformers and Its Applications (CVPR 2024 · similarity 0.85)
- Learning Linear Attention in Polynomial Time (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
