Research map / Large language models / Instruction and fine-tuning
Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VocADT, which learns a linear-combination vocabulary adapter over frozen Mistral-7B embeddings to adapt to new multilingual vocabularies, and benchmarks it against ZeTT, FOCUS, and OFA across 11 languages and MT/NLU tasks.
Problem
Vocabulary adaptation of pretrained LLMs to new languages and to reduce token over-fragmentation, where existing methods rely on heuristics or external embeddings, and their effects on generative tasks like machine translation are understudied.
Approach
Insert a learnable vocabulary adapter matrix that linearly combines original embeddings to form new embeddings, trained with a language-modeling loss while all original weights stay frozen, initialized from overlapping tokens and tokenizer partitions, and merged into a new embedding matrix after training.
Key results
- VocADT-multi improves over original Mistral on average across MT and NLU tasks after adaptation, e.g. xx-en xCOMET average 0.834 vs 0.742 for Mistral over 8 pairs.
- After full-weight ALMA-style fine-tuning, VocADT achieves the highest average MT score in en-xx (0.779 vs 0.757 for Mistral) over 8 pairs and matches ZeTT in xx-en (0.899).
- Latin-script and severely fragmented languages (e.g., Greek) benefit most from vocabulary adaptation, while Korean drops 14-15% in MT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt the vocabulary with VocADT before full-weight fine-tuning | Mistral-7B · FLORES xx-en, Avg (8 pairs) | xCOMET-XL: 0.875 -> 0.899 | Helped |
| Adapt the vocabulary with VocADT before full-weight fine-tuning | Mistral-7B · FLORES en-xx, Avg (8 pairs) | xCOMET-XL: 0.757 -> 0.779 | Helped |
| Adapt vocabulary with VocADT instead of leaving Mistral's vocabulary unchanged | Mistral-7B · FLORES xx-en, Avg (8 pairs), adaptation only | xCOMET-XL: 0.742 -> 0.834 | Helped |
| Adapt vocabulary with VocADT instead of leaving Mistral's vocabulary unchanged | Mistral-7B · FLORES en-xx, Avg (8 pairs), adaptation only | xCOMET-XL: 0.565 -> 0.637 | Helped |
| Adapt vocabulary with VocADT instead of leaving Mistral's vocabulary unchanged | Mistral-7B · XNLI, Avg (5 langs) | accuracy: 0.447 -> 0.457 | Helped |
| Adapt vocabulary with VocADT instead of leaving Mistral's vocabulary unchanged | Mistral-7B · XCOPA, Avg (4 langs) | accuracy: 0.520 -> 0.567 | Helped |
| Adapt vocabulary with VocADT instead of leaving Mistral's vocabulary unchanged | Mistral-7B · Belebele, Avg (9 langs) | accuracy: 0.581 -> 0.614 | Helped |
| Adapt vocabulary with VocADT instead of leaving Mistral's vocabulary unchanged | Mistral-7B · Multilingual MMLU, Avg (3 langs) | accuracy: 0.525 -> 0.485 | Hurt |
| Add auxiliary loss to preserve overlapping-token embeddings (ablation) | VocADT, Latin group plus English · Multilingual task performance | task performance | No clear effect |
| Add auxiliary loss to preserve overlapping-token embeddings (ablation) | VocADT, Mixed group plus English · Multilingual task performance | task performance | Helped |
| Use a consistent Cyrillic-script group rather than the Mixed-script group | VocADT, Russian · FLORES MT | xCOMET-XL | Helped |
| Use a consistent Cyrillic-script group rather than the Mixed-script group | VocADT, Bulgarian · FLORES MT | xCOMET-XL | No clear effect |
Surprising findings
- Korean declines substantially in MT (about -14% to -15%) despite reduced fragmentation, which the authors attribute to the linear-combination assumption failing for Hangul.
- The auxiliary loss hurts Latin-script languages, so alpha was set to 0 for Latin and 0.1 for non-Latin groups.
Limitations
- Linear combination assumption may not hold for non-alphabetic scripts such as Korean Hangul.
- Evaluation of MT uses xCOMET-XL at sentence level without context.
- Languages were limited to 11 and a single base model family for main results.
Experimental setup
Datasets: MADLAD-400 NLLB bitext FLORES XNLI XCOPA Belebele MMMLU
Benchmarks: FLORES XNLI XCOPA Belebele MMMLU
Metrics: xCOMET-XL accuracy
Baselines: ZeTT FOCUS OFA Mistral-7B
Builds on: ALMA FOCUS ZeTT OFA MAD-X
Closest papers by meaning
- Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies (NeurIPS 2024 · similarity 0.87)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.86)
- Enhancing Large Language Models through Adaptive Tokenizers (NeurIPS 2024 · similarity 0.86)
- Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training (NeurIPS 2025 · similarity 0.85)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.85)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.