Research map / Large language models / Language, safety and interpretability
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
Sangwhan Moon, Daisuke Oba, Youmi Ma, Tatsuya Hiraoka, Naoaki Okazaki
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evaluate on unseen 4-byte characters instead of common characters | 355M byte-fallback LM · Level 0 partial-credit validity | partial-credit validity (%): 96.21 -> 86.97 | Hurt |
| use 3-byte unseen control instead of 4-byte unseen at prefix=1 (ablation) | 355M byte-fallback LM · Level 0 unseen chars | partial-credit validity (%): 0.0 -> 48.2 | Improved |
| use 3-byte unseen control instead of 4-byte unseen at prefix=2 (ablation) | 355M byte-fallback LM · Level 0 unseen chars | partial-credit validity (%): 39.5 -> 51.1 | Improved |
| use 4-byte unseen instead of 3-byte unseen at prefix=3 (ablation) | 355M byte-fallback LM · Level 0 unseen chars | partial-credit validity (%): 19.4 -> 60.5 | Mixed |
| vary character frequency within the 3-byte class (ablation) | 355M byte-fallback LM · Level 0 | partial-credit validity: 0.878 to 0.894 | No clear effect |
| add semantic context (Level 1 vs Level 0) | 355M byte-fallback LM · Common tier | validity convergence speed | Improved |
| compare validity with perplexity convergence in Level 0 | 355M byte-fallback LM · Level 0 | convergence steps: 740 steps vs 380 steps | Mixed |
| scale from Llama-3.2 1B to 3B | Llama-3.2 · Cross-model | strict validity V_s (%): 38.4 -> 49.0 | Improved |
| scale from Qwen-3.5 4B to 9B | Qwen-3.5 · Cross-model | strict validity V_s (%): 87.0 -> 89.8 | Improved |
| scale from OLMo-2 1B to 7B | OLMo-2 · Cross-model | strict validity V_s (%): 33.5 -> 28.0 | Hurt |
| scale from Qwen-3.5 4B to 9B | Qwen-3.5 · Cross-model Level 1 | term match rate (%): 2.4 -> 0.5 | Hurt |
| use byte-fallback SentencePiece tokenizer instead of larger-vocabulary BPE | Llama-2 7B / Mistral 7B · Cross-model Level 1 | term match rate (%): 33.1 vs 0.5 | Improved |
Closest papers by meaning
- Language Models over Canonical Byte-Pair Encodings (ICML 2025 · similarity 0.85)
- Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations (NeurIPS 2025 · similarity 0.85)
- Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles (ICLR 2025 · similarity 0.85)
- From Language Models over Tokens to Language Models over Characters (ICML 2025 · similarity 0.84)
- Improving Open-Ended Text Generation via Adaptive Decoding (ICML 2024 · similarity 0.84)
- What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
