Research map / Large language models / Retrieval and knowledge
ALLaM: Large Language Models for Arabic and English
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents ALLaM, a series of Arabic-English LLMs (7B, 13B, 70B via Llama-2 continued pretraining with tokenizer expansion, and a 7B from scratch) aligned with SFT and DPO, achieving strong Arabic benchmark results.
Problem
Frontier LLMs are mostly English-centric, and Arabic has limited pretraining data and poor tokenization in existing open models; the paper asks how to add Arabic to strong models without catastrophic forgetting of English.
Approach
Second-language acquisition via tokenizer merging with an Arabic tokenizer, embedding initialization from averaged original-token embeddings, continued pretraining on an Arabic/English mix including machine-translated Arabic data, followed by SFT on Ultra-Instinct and DPO with human-verified preference pairs.
Key results
- ALLaM-70B achieves the best scores on five of eight Arabic benchmark sets, including MMLU Arabic, Exams, ETEC and AraTruthfulQA.
- On MT-Bench, ALLaM 70B Instruct scores 8.19 in Arabic, the highest among the listed models (GPT 3.5 Turbo 8.12, Command R+ 7.97).
- In human evaluation, ALLaM-13B ranks second in ELO behind GPT-4, tying with Command-R+ and outperforming larger models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use quality-filtered Ultra-Instinct v2 SFT data instead of v1 (ablation) | ALLaM 13B · SFT evaluation benchmarks | MMLU English: 63.8 -> 63.3 | No clear effect |
| use quality-filtered Ultra-Instinct v2 SFT data instead of v1 (ablation) | ALLaM 13B · Exams (ar) | score: 56.8 -> 56.8 | No clear effect |
| use quality-filtered Ultra-Instinct v2 SFT data instead of v1 (ablation) | ALLaM 13B · ACVA | score: 79.8 -> 76.7 | Hurt |
| use quality-filtered Ultra-Instinct v2 SFT data instead of v1 (ablation) | ALLaM 13B · ETEC | score: 66.8 -> 65.9 | Hurt |
| initialize expanded-token embeddings from combinations of existing embeddings rather than randomly (ablation) | ALLaM continued pretraining · Arabic language acquisition | learning speed | Helped |
| add dropout during continued pretraining (ablation) | Llama-2 language expansion · Arabic | performance | Helped |
| add dropout during continued pretraining (ablation) | Llama-2 language expansion · English | performance | Hurt |
| add machine-translated Arabic data to pretraining (ablation) | ALLaM continued pretraining · Arabic-English alignment and English retention | capability alignment / catastrophic forgetting | Helped |
| use a 45/55 Arabic/English pretraining ratio (ablation) | ALLaM continued pretraining · English and translated Arabic MMLU | performance | Helped |
| use gradual learning-rate increase and decay instead of the pretrained model's final learning rate (ablation) | ALLaM continued pretraining · original-language performance | performance retention | Hurt |
| warm up optimizer state during continued pretraining (ablation) | ALLaM continued pretraining · overall performance | performance | No clear effect |
| pretrain from scratch first on English, then on mixed Arabic/English (ablation) | ALLaM 7B from scratch · English and Arabic | benchmark performance | Helped |
Surprising findings
- Adding dropout during continued pretraining helped Arabic acquisition but hurt source-language (English) performance, so it was not used.
- Even 1% noisy SFT samples or ~0.1% noisy DPO data could noticeably degrade model quality.
Limitations
- Training data may contain toxic content and societal biases; safety training may be insufficient.
- Pretraining data is not released and no timeline is given for sharing it.
- Reliance on DPO over PPO was due to logistical constraints rather than a measured comparison.
Experimental setup
Datasets: Dolma-v1 The Pile The Stack Ultra-Instinct MT-Bench Arabic ETEC AraMath AraTruthfulQA MMLU HellaSwag ACVA Exams-Ar
Benchmarks: MMLU MMLU Arabic Exams-Ar ACVA ETEC AraTruthfulQA MT-Bench MMLU-Pro MixEval HellaSwag GSM8K HumanEval
Metrics: accuracy MT-Bench score (0-10) human win rate ELO score
Baselines: Jais-30B AceGPT 13B GPT-4 Command-R+ Llama 3-Instruct GPT 3.5 Turbo
Builds on: Llama-2 DPO Megatron-LM FlashAttention
Closest papers by meaning
- Alignment at Pre-training! Towards Native Alignment for Arabic LLMs (NeurIPS 2024 · similarity 0.90)
- L3Ms — Lagrange Large Language Models (ICLR 2025 · similarity 0.86)
- Aligner: Efficient Alignment by Learning to Correct (NeurIPS 2024 · similarity 0.86)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.86)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.86)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.