Research map / Large language models / Language, safety and interpretability
From Tokens to Words: On the Inner Lexicon of LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper shows LLMs internally detokenize multi-token words into whole-word representations at the last sub-word token (mainly early-middle layers, via FFN updates and early attention aggregation), and uses this to expand vocabulary without finetuning.
Problem
LLMs process sub-word tokens while language is composed of words; it is unclear whether and how models internally compose whole-word representations from sub-word tokens (detokenization), including for out-of-vocabulary words.
Approach
Probes, logit lens, and Patchscopes show that the last sub-word token's hidden state comes to match the full word's representation in early-middle layers, with FFN updates carrying the word concept and early attention aggregating prefix tokens; this inner lexicon is then used to add new vocabulary entries with learned linear maps and no finetuning of core parameters.
Key results
- A kNN probe on last-token hidden states distinguishes real words from nonwords with 89% accuracy at layer 13 of Llama2-7B, while the penultimate token reaches only 61%.
- For Llama2-7B, 64% of multi-token words are repeated correctly via Patchscopes from layers 5-7 (77.4% in at least one layer), with 22.6% never decoded.
- Vocabulary expansion without finetuning yields 10.5% to 14.5% token reduction across WikiText, PubMed and Arabic MWiki, with new-token accuracy of 0.402 vs 0.117 for mean embedding on Arabic MWiki.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize expanded-vocabulary entries with detokenized representations instead of mean token embeddings | Llama2-7B vocabulary expansion · WikiText, new words (new token) | token-level top-1 accuracy: 0.071 -> 0.171 | Helped |
| Initialize expanded-vocabulary entries with detokenized representations instead of mean token embeddings | Llama2-7B vocabulary expansion · PubMed, new words (new token) | token-level top-1 accuracy: 0.123 -> 0.180 | Helped |
| Initialize expanded-vocabulary entries with detokenized representations instead of mean token embeddings | Llama2-7B vocabulary expansion · mWiki-Arabic, new words (new token) | token-level top-1 accuracy: 0.117 -> 0.402 | Helped |
| Expand vocabulary using detokenized representations rather than keep the original vocabulary | Llama2-7B · WikiText, all words | token-level top-1 accuracy: 0.522 -> 0.519 | No clear effect |
| Expand vocabulary using detokenized representations rather than keep the original vocabulary | Llama2-7B · PubMed, all words | token-level top-1 accuracy: 0.517 -> 0.511 | No clear effect |
| Expand vocabulary using detokenized representations rather than keep the original vocabulary | Llama2-7B · mWiki-Arabic, all words | token-level top-1 accuracy: 0.535 -> 0.532 | No clear effect |
| Remove FFN updates carrying full-word representations (ablation) | Llama2-7B detokenization of suffix-split words · Suffix-split words | retrieval rate: 85% -> 18% | Hurt |
| Add the identified full-word FFN updates rather than randomly selected FFN updates (ablation) | Llama2-7B detokenization of suffix-split words · Suffix-split words | retrieval rate | Helped |
| Remove country-name FFN updates (ablation) | Llama2-7B capital-city retrieval from split country names · Split country names | accuracy: 88% -> 41% | Hurt |
| Remove random FFN updates (ablation) | Llama2-7B capital-city retrieval from split country names · Split country names | accuracy: 88% -> 74% | Hurt |
| Use detokenized representations rather than mean token embeddings to initialize expanded vocabulary | Llama2-7B vocabulary expansion · All words, WikiText | token-level top-1 accuracy: 0.473 -> 0.519 | Helped |
| Use detokenized representations rather than mean token embeddings to initialize expanded vocabulary | Llama2-7B vocabulary expansion · All words, PubMed | token-level top-1 accuracy: 0.479 -> 0.511 | Helped |
Surprising findings
- Ablating FFN updates carrying the full-word concept drops retrieval from 85% to 18%, while ablating random FFN updates has little effect.
- Penultimate-token probes reach only 61% accuracy, suggesting the word/nonword signal is not merely token co-occurrence.
Limitations
- Around 23% of multi-token words are never decoded from any layer, so the inner lexicon does not cover all words.
- Vocabulary expansion evaluated only on Llama2-7B and three corpora, with a short continued pretraining run for refinement matrices.
Experimental setup
Datasets: WikiText Gutenberg corpus PubMed abstracts Arabic Wikipedia (MWiki) ARC Nonword Database
Benchmarks: WikiText PubMed MWiki-Arabic ARC Nonword Database
Metrics: rate of retrieval kNN classification accuracy token-level top-1 accuracy attention weights token reduction
Baselines: Original model Mean embedding vocabulary expansion
Builds on: Patchscopes logit lens FFN key-value memories
Closest papers by meaning
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.90)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.89)
- Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.88)
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training (NeurIPS 2025 · similarity 0.88)
- Rep2Text: Decoding Full Text from a Single LLM Token Representation (NeurIPS 2026 · similarity 0.88)
- The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.