Research map / Large language models / Language, safety and interpretability
Rethinking Optimization and Architecture for Tiny Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of tiny (1B-1.5B) language model design across tokenizer compression, architecture shape, parameter inheritance from a 7B model, and multi-round training, yielding PanGu-π Pro models that beat similarly sized and some larger open models.
Problem
Tiny language models for mobile/edge devices lack carefully studied training recipes; the paper asks which design and optimization choices matter most for high performance at ~1B parameters.
Approach
Systematically ablates four components (compact tokenizer, depth-heavy architecture, parameter inheritance from PanGu-π-7B via learnable binary masks after removing redundant intermediate layers, and loss-weighted multi-round training with sample selection) and combines them into PanGu-π-1B Pro and 1.5B Pro.
Key results
- PanGu-π-1B Pro improves average score by about 8.9 over PanGu-π-1B (42.41 to 51.28 on the OpenCompass suite).
- PanGu-π-1.5B Pro reaches 60.64 average, beating Qwen-1.8B (55.04) with 16.67% fewer parameters.
- Multi-round training with 50% sampling for two rounds raises ablation average from 51.61 to 54.46 on the small-scale setup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| compress the tokenizer from 100k to 48k vocabulary (ablation) | PanGu-π-1B · ARC-E, HellaSwag, C3 average | Avg.: 42.41 to 44.11 | Helped |
| use an 8k tokenizer instead of the 48k tokenizer (ablation) | 1B model · ARC-E, HellaSwag, C3 | Avg.: 41.19 -> 37.94 | Hurt |
| increase depth to 40 layers and reduce width to 1280 (ablation) | 1B model · ARC-E, HellaSwag, C3 | Avg.: 41.19 -> 42.02 | Helped |
| increase depth to 40 layers and reduce width to 1280 (ablation) | 1B model · inference | tokens per second: 29.49 -> 12.81 | Hurt |
| set FFN expansion rate to 1.00 (ablation) | 1B model · ARC-E, HellaSwag, C3 | Avg.: 41.19 -> 37.71 | Hurt |
| inherit parameters from a larger model (ablation) | 1B model · ARC-E, HellaSwag, C3 | Avg.: 42.06 -> 48.08 | Helped |
| use learned masks to select inherited parameters instead of Taylor selection (ablation) | 1B model · ARC-E, HellaSwag, C3 | Avg.: 47.90 -> 48.08 | Helped |
| increase the second-round sampling rate from 0% to 50% (ablation) | PanGu-π-1B training · ARC-E, HellaSwag, C3 | Average: 51.61 -> 54.46 | Helped |
| increase the sampling rate from 50% to 100% (ablation) | PanGu-π-1B training · ARC-E, HellaSwag, C3 | Average: 54.46 -> 54.87 | Helped |
| train for a third round instead of two rounds (ablation) | PanGu-π-1B training · ARC-E, HellaSwag, C3 | Average: 54.46 -> 54.44 | No clear effect |
| replace MHA with GQA (ablation) | PanGu-π-1.5B Pro · C-Eval, CMMLU, MMLU, AGI-Eval | benchmark scores: 52.91 / 49.51 / 53.76 / 44.42 -> 48.75 / 46.94 / 51.97 / 43.59 | Hurt |
| use the depth-adaptive initialization method instead of constant initialization (ablation) | 1B model · ARC-E, HellaSwag, C3 | Avg.: 42.59 -> 42.94 | Helped |
Surprising findings
- Constant standard deviation random initialization performed as well as or better than GPT-2 and InternLM scaled initializations.
- Vocabulary compression to 8k degraded performance, and larger vocabularies (100k) did not help the tiny model.
Limitations
- Ablations run on reduced token budgets (5B to 50B) rather than full 1.6T training.
- Parameter inheritance depends on access to a 7B teacher model, and GQA conversion loses accuracy relative to MHA (C-Eval 52.91 to 48.75).
- Evaluation relies on OpenCompass-style benchmarks; the 6T-token result is reported only for a single extended model.
Experimental setup
Datasets: ARC Easy HellaSwag C3 RedPajama PanGu-π pretraining corpus (1.6T tokens)
Benchmarks: C-Eval CMMLU MMLU AGI-Eval BoolQ AX-b PIQA EPRSTMT XSum C3 ARC-E HellaSwag
Metrics: accuracy average accuracy tokens per second
Baselines: TinyLLaMA-1.1B Qwen-1.8B Phi2-2.7B Open-LLaMA-3B Sheared-LLaMA-1.3B MobileLLaMA-2.7B Qwen2-1.5B Qwen2.5-1.5B RWKV-5-1.5B Gemma-2.5B
Builds on: PanGu-π GPT-2 initialization LLaMA architecture GQA
Closest papers by meaning
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.89)
- Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.88)
- Cross-model Control: Improving Multiple Large Language Models in One-time Training (NeurIPS 2024 · similarity 0.88)
- An Efficient Pruner for Large Language Model with Theoretical Guarantee (ICML 2025 · similarity 0.88)
- A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (ICLR 2024 · similarity 0.88)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.