Research map / Architectures and efficiency / Attention and transformers
SpaceByte: Towards Deleting Tokenization from Large Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpaceByte, a byte-level Transformer decoder that inserts larger global blocks only after spacelike bytes, closing most of the gap to subword Transformers under compute-controlled training.
Problem
Byte-level autoregressive language models lag subword tokenized Transformers when compared under matched training and inference compute, despite avoiding tokenization's biases and vulnerabilities.
Approach
Dynamically partition bytes into patches aligned with word boundaries (a spacelike-byte rule over UTF-8) and apply larger-dimension global Transformer blocks only at patch starts, with local blocks before and after; this beats fixed-patch MegaByte-style multiscale modeling.
Key results
- On PG-19 at 1e19 training FLOPs, SpaceByte reaches 1.009 bits-per-byte versus 1.083 for MegaByte and 1.138 for the byte-level Transformer, close to the SentencePiece subword Transformer at 0.989.
- On arXiv and Github, SpaceByte achieves 0.748 and 0.500 bits-per-byte, slightly beating the SentencePiece subword Transformer (0.768 and 0.508).
- At matched ~728M inference FLOPs-per-byte and 30B bytes, SpaceByte-793M+184M gets 0.918 on PG-19 versus 0.930 for MambaByte-353M and 1.000 for MegaByte-758M+262M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dynamically sized, text-aligned patches instead of fixed-size patches (ablation) | SpaceByte · PG-19 | bits-per-byte: 1.112 -> 1.009 | Helped |
| use dynamically sized, text-aligned patches instead of fixed-size patches (ablation) | SpaceByte · arXiv | bits-per-byte: 0.804 -> 0.748 | Helped |
| use dynamically sized, text-aligned patches instead of fixed-size patches (ablation) | SpaceByte · Github | bits-per-byte: 0.552 -> 0.500 | Helped |
| replace SentencePiece subword tokenization with SpaceByte byte-level modeling | Transformer (SentencePiece) · PG-19 | bits-per-byte: 0.989 -> 1.009 | Hurt |
| replace SentencePiece subword tokenization with SpaceByte byte-level modeling | Transformer (SentencePiece) · arXiv | bits-per-byte: 0.768 -> 0.748 | Helped |
| replace SentencePiece subword tokenization with SpaceByte byte-level modeling | Transformer (SentencePiece) · Github | bits-per-byte: 0.508 -> 0.500 | Helped |
| replace GPT2 subword tokenization with SpaceByte byte-level modeling | Transformer (GPT2 tokenizer) · PG-19 | bits-per-byte: 1.013 -> 1.009 | Helped |
| replace GPT2 subword tokenization with SpaceByte byte-level modeling | Transformer (GPT2 tokenizer) · arXiv | bits-per-byte: 0.796 -> 0.748 | Helped |
| replace GPT2 subword tokenization with SpaceByte byte-level modeling | Transformer (GPT2 tokenizer) · Github | bits-per-byte: 0.554 -> 0.500 | Helped |
| use SpaceByte instead of MegaByte | MegaByte · PG-19 | bits-per-byte: 1.083 -> 1.009 | Helped |
| use SpaceByte instead of MegaByte | MegaByte · arXiv | bits-per-byte: 0.822 -> 0.748 | Helped |
| use SpaceByte instead of MegaByte | MegaByte · Github | bits-per-byte: 0.570 -> 0.500 | Helped |
Surprising findings
- Fixing SpaceByte's patch size to 6 or 8 (instead of dynamic word-aligned patches) significantly degrades performance, showing dynamic patch boundaries are crucial.
- SpaceByte performs worse than subword Transformers on Chinese text according to preliminary experiments.
Limitations
- The spacelike rule is heuristic and tailored to space-delimited text; it is not suited to images or audio.
- Performance on languages without spaces such as Chinese is worse than subword Transformers.
- Variable spacing between global blocks complicates efficient batched inference sampling.
Experimental setup
Datasets: PG-19 arXiv (The Pile) Github (The Pile) Stories
Benchmarks: PG-19 arXiv Github Stories
Metrics: bits-per-byte FLOPs-per-byte training FLOPs
Baselines: MegaByte MambaByte-353M Transformer (GPT2 tokenizer) Transformer (SentencePiece) Transformer (Window Attention) PerceiverAR-248M Transformer-320M
Builds on: MegaByte MambaByte Hourglass Transformer RoPE FlashAttention
Closest papers by meaning
- MrT5: Dynamic Token Merging for Efficient Byte-level Language Models (ICLR 2025 · similarity 0.88)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.88)
- Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models (ICLR 2025 · similarity 0.88)
- From Bytes to Ideas: Language Modeling with Autoregressive U-Nets (NeurIPS 2025 · similarity 0.88)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.88)
- From Language Models over Tokens to Language Models over Characters (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.