Research map / Architectures and efficiency / Attention and transformers
General-purpose, long-context autoregressive modeling with Perceiver AR
Curtis Hawthorne, Andrew Jaegle, Cătălina Cangea, Sebastian Borgeaud, Charlie Nash, Mateusz Malinowski, Sander Dieleman, Oriol Vinyals, Matthew Botvinick, Ian Simon, Hannah Sheahan, Neil Zeghidour and 3 more
This paper tried a common change: Larger batch size
Increase batch size from 64 to 128 while using 1024 latents · Copy task, context length 8192 · test accuracy: <1% -> 100% Improved
What 785 accepted papers report for this change
- 469 improved
- 107 no clear effect
- 48 mixed
- 161 hurt
- Predicting the Emergence of Induction Heads in Language Model Pretraining ICML 2026 · increase batch size (fixed context 1024) · Natural data (CC100) · induction head emergence point (updates): rho=-1.0, p<0.001Improved
- When RL Meets Adaptive Speculative Training: A Unified Training-Serving System ICML 2026 · Increase batch size from 1 to 32 · MiniMax M2.1 FP8 · speedup: 1.57x -> 1.25xHurt
- Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking ICML 2026 · increase per-device batch size to 10 using freed memory · Llama 3-8B, 128K · throughput (TPS): 2464.10 (Ulysses) -> 2526.77 (UPipe)Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Perceiver AR for long-context autoregressive modeling | ImageNet 64×64 density estimation · ImageNet 64×64 val | bits/dim: 3.40 versus 3.42 | Improved |
| Use Perceiver AR with 2048 context instead of Transformer-XL | PG-19 language modeling · PG-19 test | perplexity: 36.3 -> 28.9 | Improved |
| Extend context to 16384 in Perceiver AR | Books language modeling · Books test, compute-matched | perplexity: 14.276 -> 13.749 | Improved |
| Increase evaluation latents from 1024 to 1536 (ablation) | Perceiver AR · ImageNet 64×64 val | bits/dim: 3.4025 -> 3.3986 | Improved |
| Increase evaluation latents from 1024 to 4096 (ablation) | Perceiver AR · ImageNet 64×64 val | bits/dim: 3.4025 -> 3.5569 | Hurt |
| Reduce evaluation latents from 1024 to 16 (ablation) | Perceiver AR · ImageNet 64×64 val | bits/dim: 3.4025 -> 3.5664 | Hurt |
| Increase input context from 1024 to 12289 (ablation) | Perceiver AR with R→G→B ordering · ImageNet 64×64 val | bits/dim: 4.63 -> 3.53 | Improved |
| Increase input context from 1024 to 12289 (ablation) | Perceiver AR with standard ordering · ImageNet 64×64 val | bits/dim: 3.55 -> 3.54 | No clear effect |
| Increase context from 1024 to 16384 (ablation) | Perceiver AR · Books eval | perplexity: 14.88 -> 14.56 | Improved |
| Increase context from 1024 to 2048 (ablation) | Perceiver AR · Wikitext-103 test | perplexity: 18.52 -> 18.35 | Improved |
| Increase context from 2048 to 4096 (ablation) | Perceiver AR · Wikitext-103 test | perplexity: 18.35 -> 18.25 | Improved |
| Increase context from 2048 to 8192 (ablation) | Perceiver AR · Wikitext-103 test | perplexity: 18.35 -> 18.37 | Hurt |
Closest papers by meaning
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.87)
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.86)
- Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots (ICML 2025 · similarity 0.86)
- Autoregressive Pretraining with Mamba in Vision (ICLR 2025 · similarity 0.86)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.86)
- JetFormer: An autoregressive generative model of raw images and text (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
