Research map / Large language models / Language, safety and interpretability
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BigScience Workshop
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multilingual byte-level BPE tokenizer instead of monolingual tokenizer | BLOOM tokenizer · Universal Dependencies 2.9 (fr) | fertility: 1.30 -> 1.17 (-11%) | Improved |
| use multilingual byte-level BPE tokenizer instead of monolingual tokenizer | BLOOM tokenizer · Universal Dependencies 2.9 (ar) | fertility: 1.16 -> 1.34 (+13%) | Hurt |
| use multilingual byte-level BPE tokenizer instead of monolingual tokenizers | BLOOM tokenizer · Universal Dependencies 2.9 (6 languages) | fertility: fr -11%, en +0%, es +3%, zh +5%, hi +9%, ar +13% | Mixed |
| add unicode NFKC normalization to the tokenizer (ablation) | BLOOM tokenizer · fertility on considered languages | fertility: did not reduce fertility by more than 0.8% | No clear effect |
| use causal decoder-only architecture instead of encoder-decoder or other objectives (ablation) | pretraining architecture · zero-shot generalization (EAI-Eval, T0-Eval) | zero-shot performance | Improved |
| use ALiBi positional embeddings instead of learned or rotary embeddings (ablation) | BLOOM (1.3B ablation) · zero-shot downstream tasks | downstream performance and training smoothness | Improved |
| add embedding LayerNorm after the embedding layer (ablation) | BLOOM 104B preliminary · training stability and zero-shot generalization | stability / zero-shot performance | Mixed |
| deduplicate lines in tokenizer training data (ablation) | BLOOM tokenizer · tokenizer vocabulary quality | vocabulary quality | Improved |
| use a 250k vocabulary instead of 150k (ablation) | BLOOM tokenizer · fertility vs monolingual tokenizers | fertility | Improved |
| train in bfloat16 instead of float16 | BLOOM 176B · training stability | stability | Improved |
Later papers that compared with this method
3 later accepted papers report BLOOM 560M as a baseline in their results tables; 2 of them report a better number on most of the results they share with it.
- Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training (ICLR 2026 · better on 1 of 3 shared results)
- PonderLM: Pretraining Language Models to Ponder in Continuous Space (ICLR 2026 · better on 1 of 1 shared result)
- The Same but Different: Structural Similarities and Differences in Multilingual Language Modeling (ICLR 2025 · better on 2 of 3 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.88)
- OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models (ICML 2024 · similarity 0.88)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.88)
- Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining (NeurIPS 2025 · similarity 0.87)
- The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model (ICLR 2025 · similarity 0.87)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
