AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture

Shuchen Xue, Tianyu Xie, Tianyang Hu, Zijin Feng, Jiacheng Sun, Kenji Kawaguchi, Zhenguo Li, Zhi-Ming Ma

ICML 2026 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add 10% left-to-right data to any-order training (ablation)AO-GPT · LAMBADA (medium, left-to-right evaluation)zero-shot perplexity: 61.05 -> 42.44Improved
add 10% left-to-right data to any-order training (ablation)AO-GPT · PTB (medium, any-order evaluation)zero-shot perplexity: 126.11 -> 101.01Improved
ensemble over 64 context permutations at evaluation (ablation)AO-GPT · LAMBADA (medium, any-order evaluation)zero-shot perplexity: 49.79 -> 44.31Improved
ensemble over 8 context permutations at evaluation (ablation)AO-GPT · LAMBADA (medium, any-order evaluation)zero-shot perplexity: 49.79 -> 45.08Improved
ensemble over 64 context permutations at evaluation (ablation)AO-GPT · PTB (small, order-agnostic evaluation)zero-shot perplexity: 141.92 -> 123.49Improved
use decoder-only AO-GPT instead of encoder-only RADDAO-GPT · LAMBADA (medium, any-order evaluation)zero-shot perplexity: 41.96 -> 49.79Hurt
use decoder-only AO-GPT instead of encoder-only RADDAO-GPT · PTB (medium, left-to-right evaluation)zero-shot perplexity: 76.00 -> 87.56Hurt
anneal sampling (top-p 0.95, temp 0.7) instead of no annealingAO-GPT-Medium · OpenWebText generation, 1024 stepsgeneration perplexity: 136.4 -> 4.6Improved
compare AO-GPT against SEDD without annealingAO-GPT-Medium · OpenWebText generation, 1024 steps (top-p 1.0)generation perplexity: 81.6 -> 136.4Hurt
use AO-GPT instead of SEDD for generation speedAO-GPT · generation speedspeedup: ~25x speedupImproved
train with any-order instead of left-to-right order (ablation)GPT-2 small decoder · OpenWebText trainingvalidation loss: slower convergenceHurt
add EMA to weights (decay 0.9999) (ablation)AO-GPT · OpenWebText trainingtraining loss: all EMA decays 0.99, 0.999, 0.9999 beat no EMAImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy