Research map / Architectures and efficiency / Attention and transformers
ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard, Besnik Fetahu, Nasser Zalmout, Xian Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use lossy coding-rate Top-K chunking instead of cosine-similarity chunking (ablation) | ByteFlow Net (600M) · Chunking ablation | validation BPB: 0.92 -> 0.86 | Improved |
| use coding-rate chunking instead of word-boundary chunking (ablation) | ByteFlow Net (600M) · Chunking ablation | task accuracy: 49.38 -> 50.89 | Improved |
| use random chunk boundaries (ablation) | ByteFlow Net (600M) · Chunking ablation | task accuracy: 50.89 -> 41.34 | Hurt |
| use fixed-stride chunking (ablation) | ByteFlow Net (600M) · Chunking ablation | validation BPB: 0.86 -> 0.96 | Hurt |
| use word-boundary chunking instead of LLaMA BPE tokenization (ablation) | byte-level hierarchical model (600M) · zero-shot task accuracy | task accuracy: 49.15 -> 49.38 | No clear effect |
| add Canon layers to local encoder/decoder (ablation) | ByteFlow Net (600M) · Zero-shot average (600M) | accuracy: 49.04 -> 50.89 | Improved |
| add Canon layers to local encoder/decoder (ablation) | ByteFlow Net (1.3B) · Zero-shot average (1.3B) | accuracy: 61.06 -> 63.19 | Improved |
| increase global sequence length (lower compression) from 1600 to 4096 (ablation) | ByteFlow Net (600M) · Zero-shot average (600M) | accuracy: 48.48 -> 51.74 | Improved |
| increase global sequence length from 1600 to 4096 (ablation) | ByteFlow Net (1.3B) · Zero-shot average (1.3B) | accuracy: 60.91 -> 64.44 | Improved |
| replace BPE LLaMA with ByteFlow Net at 600M | LLaMA · Zero-shot average (600M) | accuracy: 49.15 -> 50.89 | Improved |
| replace BPE LLaMA with ByteFlow Net at 1.3B | LLaMA · Zero-shot average (1.3B) | accuracy: 60.15 -> 63.19 | Improved |
| replace Llama 3 with ByteFlow Net 1.3B | LLM · CUTE | accuracy: 27.5 -> 51.2 | Mixed |
Closest papers by meaning
- MrT5: Dynamic Token Merging for Efficient Byte-level Language Models (ICLR 2025 · similarity 0.89)
- zip2zip: Inference-Time Adaptive Tokenization via Online Compression (NeurIPS 2025 · similarity 0.89)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.88)
- From Bytes to Ideas: Language Modeling with Autoregressive U-Nets (NeurIPS 2025 · similarity 0.88)
- MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization (NeurIPS 2024 · similarity 0.87)
- SpaceByte: Towards Deleting Tokenization from Large Language Modeling (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
