Research map / Large language models / Agents, code and math
StochasTok: Improving Fine-Grained Subword Understanding in LLMs
Anya Sims, Thomas Foster, T. Duy Nguyen-Hien, Klara Kaleb, Joseph Lee, Jakob Nicolaus Foerster, Yee Whye Teh, Cong Lu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain with StochasTok instead of deterministic BPE | 50M GPT-2 BPE model · LangGame (finetuned with BPE) | accuracy | Improved |
| pretrain with StochasTok instead of BPE-dropout (ablation) | 50M model · LangGame | accuracy | Improved |
| pretrain with StochasTok instead of deterministic BPE | 50M model · CUTE | normalized accuracy | Improved |
| pretrain with StochasTok | 50M model · standard language understanding benchmarks | accuracy | No clear effect |
| vary StochasTok p from 0.01 to 0.2 (ablation) | 50M model · LangGame | accuracy | No clear effect |
| pretrain with StochasTok on larger 275M model | modded-nanogpt 275M · LangGame | accuracy | Improved |
| pretrain with StochasTok instead of BPE or BPE-dropout | 50M model · multi-digit addition | accuracy | Improved |
| continued pretraining 2k steps with StochasTok vs deterministic BPE | pretrained 50M model · LangGame | accuracy | Improved |
| 7k steps CPT with StochasTok vs deterministic BPE | GPT-2 · LangGame | accuracy | Improved |
| pretrain with StochasTok | 50M model · OOD LangGame holdout | accuracy | Improved |
Closest papers by meaning
- From Tokens to Words: On the Inner Lexicon of LLMs (ICLR 2025 · similarity 0.89)
- The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.88)
- Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models (ICLR 2025 · similarity 0.88)
- Not All Tokens Are What You Need for Pretraining (NeurIPS 2024 · similarity 0.88)
- Enhancing LLM’s Cognition via Structurization (NeurIPS 2024 · similarity 0.88)
- Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
