Research map / Large language models / Language, safety and interpretability
MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Designs sub-billion LLMs (MobileLLM) via deep-thin architecture, embedding sharing, GQA, and immediate block-wise layer sharing (MobileLLM-LS), trained from scratch on 1T tokens and evaluated on common-sense, QA, chat, and API-calling tasks.
Problem
Building high-quality LLMs under 1B parameters for on-device deployment, where memory (DRAM) and energy constraints limit model size.
Approach
Shows that depth matters more than width for small LLMs, and combines embedding sharing, grouped-query attention, and immediate block-wise weight sharing (repeating adjacent blocks so weights stay in cache) to improve accuracy at fixed model size and minimal latency cost.
Key results
- MobileLLM-125M averages 46.3 on zero-shot common sense vs 43.6 for RWKV-169M; MobileLLM-LS-125M reaches 47.0.
- MobileLLM-350M averages 51.3 and MobileLLM-LS-350M 52.1 on zero-shot common sense, beating prior 350M SOTA by more than 4 points.
- MobileLLM-350M achieves 65.3 intent EM in API calling, comparable to LLaMA-v2 7B (62.8), though lower Rouge scores.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace vanilla FFN with SwiGLU (ablation) | 125M MobileLLM design · Zero-shot common-sense reasoning | average accuracy: 42.6 -> 43.9 | Helped |
| Use a deep-thin architecture instead of a shallower, wider structure (ablation) | 125M MobileLLM design · Zero-shot common-sense reasoning | average accuracy: 43.9 -> 44.8 | Helped |
| Share input and output embeddings (ablation) | 125M MobileLLM design · Zero-shot common-sense reasoning | average accuracy: 44.8 -> 44.6 | Hurt |
| Use grouped-query attention and increase embedding dimension (ablation) | 125M MobileLLM design · Zero-shot common-sense reasoning | average accuracy: 44.6 -> 45.0 | Helped |
| Add immediate block-wise layer sharing (ablation) | MobileLLM-125M · Zero-shot common-sense reasoning | average accuracy: 46.3 -> 47.0 | Helped |
| Add immediate block-wise layer sharing (ablation) | MobileLLM-350M · Zero-shot common-sense reasoning | average accuracy: 51.3 -> 52.1 | Helped |
| Replace immediate block-wise sharing with repeat-all-over sharing (ablation) | 125M model · Zero-shot common-sense reasoning | average accuracy: 44.6 -> 45.2 | Helped |
| Increase layer-sharing repetition from ×2 to ×4 (ablation) | 125M model · Zero-shot common-sense reasoning | average accuracy: 45.0 -> 45.3 | Helped |
| Apply W8A8 post-training quantization (ablation) | MobileLLM-125M · Zero-shot common-sense reasoning | average accuracy: 45.0 -> 44.8 | Hurt |
| Add LLaMA-v2 7B knowledge distillation loss (ablation) | 125M model trained with labels · Zero-shot common-sense reasoning | average accuracy: 43.9 -> 43.8 | Hurt |
| Add LLaMA-v2 7B knowledge distillation loss (ablation) | 350M model trained with labels · Zero-shot common-sense reasoning | average accuracy: 49.1 -> 48.8 | Hurt |
| Train MobileLLM-1.5B rather than Qwen1.5-1.8B | Zero-shot common-sense reasoning · Zero-shot common-sense reasoning, 1.5B scale | average accuracy: 56.5 -> 59.4 | Helped |
Surprising findings
- Deeper-thinner models (30-42 layers) outperform 12-layer models at ~125M parameters, contrary to scaling-law belief that architecture matters little.
- Knowledge distillation from LLaMA-v2 7B gave comparable or inferior accuracy to hard-label training while slowing training 2.6-3.2x.
Limitations
- Pretraining data is not described in detail and the API-calling dataset is synthetic and unreleased.
- Knowledge distillation was found not to help, and the study is limited to scales up to 350M in main experiments (larger sizes in appendix).
Experimental setup
Datasets: pretraining corpus (1T tokens, unspecified) synthetic API calling dataset (5000 train / 2500 test) AlpacaEval MT-Bench
Benchmarks: ARC-easy ARC-challenge BoolQ PIQA SIQA HellaSwag OBQA WinoGrande TriviaQA RACE AlpacaEval MT-Bench
Metrics: zero-shot accuracy F1 win rate MT-Bench score exact match Rouge latency
Baselines: OPT-125M GPT-neo-125M Pythia-160M RWKV-169M OPT-350M Pythia-410M BLOOM-560M LLaMA-v2 7B Qwen1.5-1.8B MobiLlama TinyLlama-1.1B
Builds on: SwiGLU grouped-query attention OPT embedding sharing LLaMA-v2
Closest papers by meaning
- PALMBENCH: A COMPREHENSIVE BENCHMARK OF COMPRESSED LARGE LANGUAGE MODELS ON MOBILE PLATFORMS (ICLR 2025 · similarity 0.89)
- BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices (CVPR 2025 · similarity 0.89)
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.88)
- AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment (NeurIPS 2024 · similarity 0.87)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.87)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.