Research map / Architectures and efficiency / Attention and transformers
Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper's tool call was malformed in the previous turn; this is a corrected call.
Problem
Subword tokenizers have large vocabularies, are fitted separately from the model, adapt poorly to new domains or languages, and are sensitive to spelling variations.
Approach
A generative hierarchical architecture that splits text at whitespace into words, encodes each word's characters with a small bidirectional encoder (taking the [W] token output as word embedding), runs a causal word-level backbone, and decodes the next word's characters from a predictive word embedding via a small causal character decoder, trained end-to-end without a tokenizer.
Key results
- At 7B compute-matched scale, the hierarchical model matches the baseline on most zero-shot tasks and beats it on LAMBADA by up to 68% relative (at 7B LBD OAI 73.8 vs 67.9).
- Under input perturbations the baseline suffers about 3x larger accuracy drop on all-caps prompts than the hierarchical model.
- During German continued pretraining the hierarchical model trains 1.9x faster in compute per step and achieves higher German and English eval scores than the tokenizer baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use hierarchical whitespace-based processing instead of the tokenizer baseline | 1B language model · DCLM-Baseline | word accuracy: 35.3 -> 35.5 | No clear effect |
| use hierarchical whitespace-based processing instead of the tokenizer baseline | 3B language model · LAMBADA (LBD) | accuracy: 62.1 -> 64.2 | Helped |
| use hierarchical whitespace-based processing instead of the tokenizer baseline | 7B language model · DCLM-Baseline | word accuracy: 39.2 -> 39.0 | No clear effect |
| replace MegaByte with the proposed hierarchical architecture while keeping the 8-byte split (ablation) | 1B byte-level language model · DCLM-Baseline | byte accuracy: 80.0 -> 82.6 | Helped |
| use whitespace splitting instead of fixed 8-byte splitting (ablation) | Hierarchical language model · LAMBADA (LBD), 1B | accuracy: 48.1 -> 56.5 | Helped |
| replace whitespace splitting with Unicode splitting (ablation) | 3B hierarchical language model · DCLM-Baseline | MMLU: 28.7 -> 31.0 | Helped |
| replace whitespace splitting with Unicode splitting (ablation) | 3B hierarchical language model · DCLM-Baseline | TruthfulQA: 29.9 -> 27.8 | Hurt |
| perturb prompts by permuting, randomizing, or deleting 10% of characters per word, or changing prompts to all caps | Hierarchical model vs tokenizer baseline · MMLU, OpenBookQA, ARC Challenge, HellaSwag, and one additional eval task | accuracy change | Helped |
| continue pretraining on German Occiglot data | 3B hierarchical model vs 3B tokenizer baseline · German and English evaluations | average accuracy: 1.9 times faster | Helped |
| continue pretraining on Chinese SkyPile data using the Unicode splitter | 3B hierarchical model vs tokenizer baseline · SkyPile held-out | bits per byte: 0.94 -> 0.80 | Helped |
| switch the splitting rule from whitespace to Unicode at continued pretraining (ablation) | 3B hierarchical model · Chinese SkyPile | bits per byte: 0.80 -> 0.80 | No clear effect |
| evaluate pretrained models on out-of-distribution text without additional training | Hierarchical model vs tokenizer baseline · OpenWebMath | bits per byte: 0.81 -> 0.81 | No clear effect |
Surprising findings
- Byte accuracy and word accuracy favor different encoder/decoder sizes, so byte accuracy was judged a poor guide for architecture selection.
- On out-of-distribution data (GitHub code, OpenWebMath, Pile of Law, SkyPile) bits-per-byte of the two models is nearly identical despite large FLOP differences.
Limitations
- Whitespace splitting is tailored to alphabetic languages and may be suboptimal for Chinese, code, or math; a Unicode splitter is only explored in appendix.
- For compute-matched models the hierarchical architecture has a larger parameter footprint (e.g., 9.2B backbone-based model vs 7.0B baseline).
Experimental setup
Datasets: DCLM-Baseline Fineweb Occiglot Fineweb v0.5 German SkyPile GitHub Code OpenWebMath Pile of Law
Benchmarks: MMLU LAMBADA ARC OpenBookQA TriviaQA TruthfulQA Winogrande HellaSwag WiC WebQs PIQA BoolQ
Metrics: zero-shot accuracy word-level accuracy byte-level accuracy bits per byte change in accuracy under perturbation FLOP ratio
Baselines: Llama-style BPE tokenizer baseline (64k vocab) MegaByte Hierarchical (8-byte split) ablation
Builds on: Llama MegaByte Hierarchical byte-level BERT (Hourglass-style character-word model) flash attention
Closest papers by meaning
- From Bytes to Ideas: Language Modeling with Autoregressive U-Nets (NeurIPS 2025 · similarity 0.92)
- The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.89)
- An Analysis of Tokenization: Transformers under Markov Data (NeurIPS 2024 · similarity 0.89)
- Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flows (NeurIPS 2025 · similarity 0.89)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.88)
- SpaceByte: Towards Deleting Tokenization from Large Language Modeling (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.