Research map / Large language models / Language, safety and interpretability
From Bytes to Ideas: Language Modeling with Autoregressive U-Nets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AU-Net, an autoregressive U-Net that learns its own token embeddings from raw bytes via multi-stage adaptive pooling at word boundaries, and matches strong BPE Transformers under equal compute while improving multilingual and character-level performance.
Problem
Fixed tokenization freezes the granularity at which language models operate and how far ahead they predict; BPE embeddings are opaque lookup tables with a fixed vocabulary.
Approach
An autoregressive U-Net that pools byte-level representations at word boundaries and word groups (rule-based regex splits), so deeper stages operate on coarser sequences and predict several words ahead, with multi-linear position-specific upsampling and skip connections, removing the need for a predefined vocabulary.
Key results
- At 1B scale on 370B tokens, AU-Net 4 reaches 73.7 Hellaswag and 31.7 MMLU versus 70.2 and 27.0 for the BPE Transformer at 4e21 FLOPs (AU-Net 4 uses 5e21 FLOPs).
- At 8B scale on 200B tokens, AU-Net 2 (7.9B) reaches 79.1 Hellaswag and 80.0 ARC-E versus 77.2 and 74.5 for the 7.5B BPE Transformer, though BPE is better on ARC-C, TQA and GSM8K.
- On FLORES-200 English-to-low-resource translation and MMLU, AU-Net 2 averages 24.6 vs 20.9 BLEU for Lang-to-Eng and 32.4 vs 31.4 MMLU average across 26 languages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase hierarchy from 2 to 3 or 4 stages | AU-Net · DCLM, 370B tokens | Hellaswag accuracy: 70.2 -> 72.9; 73.7 | Helped |
| use deeper hierarchy (3–4 stages) instead of BPE | AU-Net · DCLM, 370B tokens | ARC-E accuracy: 68.6 -> 72.3; 72.6 | Helped |
| use AU-Net 3 or AU-Net 4 instead of BPE | AU-Net · DCLM, 370B tokens | ARC-C accuracy: 38.5 -> 43.3; 43.2 | Helped |
| use AU-Net 4 instead of BPE | AU-Net · DCLM, 370B tokens | MMLU accuracy: 27.0 -> 31.7 | Helped |
| use AU-Net instead of BPE | AU-Net · DCLM, 370B tokens | GSM8K accuracy: 4.4 -> 3.0; 3.7; 5.3 | Mixed |
| use byte-level AU-Net 2 instead of BPE | AU-Net 2 · CUTE character tasks | exact match: 39.3 -> 42.7 | Helped |
| use byte-level AU-Net 2 instead of BPE | AU-Net 2 · CUTE word tasks | exact match: 36.9 -> 33.1 | Hurt |
| use byte-level AU-Net 2 instead of BPE | AU-Net 2 · FLORES-200, Lang. → Eng. | BLEU: 20.9 -> 24.6 | Helped |
| use byte-level AU-Net 2 instead of BPE | AU-Net 2 · MMLU, non-English average | exact match: 31.4 -> 32.4 | Helped |
| replace simple pooling with average pooling (ablation) | AU-Net 2 · 1B, 60B tokens; Hellaswag | accuracy: 62.9 ± 0.9 -> 62.5 ± 1.0 | No clear effect |
| add a memory layer after pooling (ablation) | AU-Net 2 · 1B, 60B tokens; ARC-E | accuracy: 64.9 ± 1.9 -> 66.5 ± 1.9 | Helped |
| replace multi-linear with repeat upsampling (ablation) | AU-Net 2 · 1B, 60B tokens; Hellaswag | accuracy: 63.5 ± 0.9 -> 64.2 ± 0.9 | Helped |
Surprising findings
- BPE Transformers still perform well on spelling and reverse-spelling CUTE tasks despite lacking explicit character access.
- Byte-level models match BPE on Hellaswag and ARC-Easy despite using much more compute in some configurations, and AU-Net underperforms BPE on Chinese MMLU (28.0 vs 33.0).
Limitations
- Relies on a predefined, space-based splitting function and does not support non-space-delimited languages such as Chinese.
- Pretraining corpus DCLM is English-dominant with little math, leading to weak GSM8K results.
- Underperforms BPE on TQA at smaller scales and on word-level CUTE tasks; FSDP overlap struggles at deeper stages.
Experimental setup
Datasets: DCLM FLORES-200 MMLU (multilingual) CUTE
Benchmarks: Hellaswag ARC-Easy ARC-Challenge MMLU NQ TQA GSM8K CUTE FLORES-200 PIQA SIQA RACE
Metrics: accuracy exact match BLEU bytes per second per GPU bootstrap 95% confidence intervals
Baselines: Transformer BPE (LLaMa 3 tokenizer) Transformer bytes Mamba bytes MegaByte Hierarchical BLT DCLM-7B LLaMa 3.1 8B
Builds on: U-Net Megabyte Byte Latent Transformer Hoffmann-style compute-optimal scaling Complete-P / hyperparameter transfer scaling
Closest papers by meaning
- Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models (ICLR 2025 · similarity 0.92)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.89)
- SpaceByte: Towards Deleting Tokenization from Large Language Modeling (NeurIPS 2024 · similarity 0.88)
- Language Models over Canonical Byte-Pair Encodings (ICML 2025 · similarity 0.88)
- The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.88)
- MrT5: Dynamic Token Merging for Efficient Byte-level Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.