Research map / Large language models / Language, safety and interpretability
Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model
Sebastian J. Mielke, Jason Eisner
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine type-level spelling training and UNK-token spelling training in one model (ablation) | Spell Once · WikiText-2 | bits per character: 1.455 vs 1.462 | Improved |
| Add spelling training on UNK tokens (ablation) | Spell Once · WikiText-2 | bits per character: 1.455 vs 1.494 | Improved |
| Share the spelling model between in-vocabulary types and UNK tokens (ablation) | Spell Once · WikiText-2 | bits per character: 1.455 vs 1.469 | No clear effect |
| Condition the spelling RNN on the word embedding (ablation) | Spell Once · WikiText-2 | bits per character: 1.468 -> 1.455 | Improved |
| Use an RNN spelling model instead of a character unigram model (ablation) | Spell Once · WikiText-2 | bits per character: 1.548 -> 1.455 | Improved |
| Use Spell Once instead of pure BPE | Language modeling · WikiText-2 | bits per character: 1.468 -> 1.455 | Improved |
| Use Spell Once instead of pure BPE | MWC English · tokenize test | bits per character: 1.386 vs 1.387 | No clear effect |
| Use Spell Once instead of pure BPE | MWC French · tokenize test | bits per character: 1.317 vs 1.319 | No clear effect |
| Use Spell Once instead of pure BPE | MWC German · tokenize test | bits per character: 1.414 -> 1.465 | Hurt |
| Use Spell Once instead of pure BPE | MWC Spanish · tokenize test | bits per character: 1.362 vs 1.363 | No clear effect |
| Use Spell Once instead of pure BPE | MWC Czech · tokenize test | bits per character: 1.856 -> 1.928 | Hurt |
| Use Spell Once instead of pure BPE | MWC Finnish · tokenize test | bits per character: 1.652 -> 1.751 | Hurt |
Closest papers by meaning
- Learning Semantic Representations for Novel Words: Leveraging Both Form and Context (AAAI 2019 · similarity 0.87)
- Towards a theory of how the structure of language is acquired by deep neural networks (NeurIPS 2024 · similarity 0.86)
- From Tokens to Words: On the Inner Lexicon of LLMs (ICLR 2025 · similarity 0.85)
- Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models (ICLR 2025 · similarity 0.85)
- Language Model Cascades: Token-Level Uncertainty And Beyond (ICLR 2024 · similarity 0.85)
- By Tying Embeddings You Are Assuming the Distributional Hypothesis (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
