Research map / Architectures and efficiency / RNNs and normalization
Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
Shuchen Xue, Tianyu Xie, Tianyang Hu, Zijin Feng, Jiacheng Sun, Kenji Kawaguchi, Zhenguo Li, Zhi-Ming Ma
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 10% left-to-right data to any-order training (ablation) | AO-GPT · LAMBADA (medium, left-to-right evaluation) | zero-shot perplexity: 61.05 -> 42.44 | Improved |
| add 10% left-to-right data to any-order training (ablation) | AO-GPT · PTB (medium, any-order evaluation) | zero-shot perplexity: 126.11 -> 101.01 | Improved |
| ensemble over 64 context permutations at evaluation (ablation) | AO-GPT · LAMBADA (medium, any-order evaluation) | zero-shot perplexity: 49.79 -> 44.31 | Improved |
| ensemble over 8 context permutations at evaluation (ablation) | AO-GPT · LAMBADA (medium, any-order evaluation) | zero-shot perplexity: 49.79 -> 45.08 | Improved |
| ensemble over 64 context permutations at evaluation (ablation) | AO-GPT · PTB (small, order-agnostic evaluation) | zero-shot perplexity: 141.92 -> 123.49 | Improved |
| use decoder-only AO-GPT instead of encoder-only RADD | AO-GPT · LAMBADA (medium, any-order evaluation) | zero-shot perplexity: 41.96 -> 49.79 | Hurt |
| use decoder-only AO-GPT instead of encoder-only RADD | AO-GPT · PTB (medium, left-to-right evaluation) | zero-shot perplexity: 76.00 -> 87.56 | Hurt |
| anneal sampling (top-p 0.95, temp 0.7) instead of no annealing | AO-GPT-Medium · OpenWebText generation, 1024 steps | generation perplexity: 136.4 -> 4.6 | Improved |
| compare AO-GPT against SEDD without annealing | AO-GPT-Medium · OpenWebText generation, 1024 steps (top-p 1.0) | generation perplexity: 81.6 -> 136.4 | Hurt |
| use AO-GPT instead of SEDD for generation speed | AO-GPT · generation speed | speedup: ~25x speedup | Improved |
| train with any-order instead of left-to-right order (ablation) | GPT-2 small decoder · OpenWebText training | validation loss: slower convergence | Hurt |
| add EMA to weights (decay 0.9999) (ablation) | AO-GPT · OpenWebText training | training loss: all EMA decays 0.99, 0.999, 0.9999 beat no EMA | Improved |
Closest papers by meaning
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.90)
- Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions (ICML 2025 · similarity 0.90)
- Looped Diffusion Language Models (NeurIPS 2026 · similarity 0.90)
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.90)
- Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.90)
- SimSD: Simple Speculative Decoding in Diffusion Language Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
