AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Learning Pseudorandom Numbers with Transformers: Permuted Congruential Generators, Curricula, and Interpretability

Tao Tao, Maissam Barkeshli

ICLR 2026 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
initialize from a smaller-modulus model (pretrained init) (ablation)Transformer on XSLRR-20/10 · XSLRR-20/10 (m=2^20)test accuracy at 640th token: 4% (random init) -> 0.9697 (pretrained+curriculum)Improved
add curriculum mixing of smaller-modulus data (ablation)Transformer on XSLRR large moduli · XSLRR-18/9 and 20/10test accuracy / stabilityImproved
mix 1% small-modulus data via curriculum instead of fixed alpha (ablation)XSLRR-18/9 · XSLRR-18/9accuracy at 256th tokenImproved
exponential decay of alpha rather than cosine, linear or step (ablation)curriculum schedule · XSLRR-18/9accuracyImproved
add curriculum (ablation)training at m=2^18 · XSLRR-18/9stable learning rate rangeImproved
mix XSLRR-18/9 data instead of XSLRR-16/8 in curriculum for m=2^20 (ablation)curriculum · XSLRR-20/10accuracyNo clear effect
increase model depth/size (ablation)XSLRR-16/8 Transformer · XSLRR-16/8test loss/accuracy: 6-layer reaches 22.8% at 64th tokenImproved
increase number of (a,c) pairs beyond 1024x1024 (ablation)XSLRR-16/8 · XSLRR-16/8test lossNo clear effect
truncate output to fewer bits (k=1) (ablation)truncated LCG · TLCG m=2^16accuracy: 95% at 256th element, lower at 64thHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy