Research map / Architectures and efficiency / RNNs and normalization
Scaling Inference-Efficient Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Modifies Chinchilla scaling laws with an aspect-ratio term to co-optimize parameters, tokens and model shape for inference latency, and releases the Morph-1B model that is 1.8x faster than comparable open models at similar accuracy.
Problem
Existing scaling laws ignore inference cost; model architecture (not just parameter count) strongly affects inference latency, and lower training loss does not reliably translate into downstream accuracy.
Approach
Extend the Chinchilla loss with a multiplicative aspect-ratio (d_model / n_layers) term so that loss depends on size, tokens and shape; then use latency measurements plus predicted loss to shortlist top-k candidate architectures, train them, and select by downstream performance and latency.
Key results
- Inference-efficient scaling law reduces loss-prediction MSE from 0.0033 to 0.0006 and raises R^2 from 0.9895 to 0.9982 versus Chinchilla, with relative error under 1.2%.
- Morph-1B improves inference latency by 1.8x over Morph-1B-v1 at the same 30B tokens and similar average accuracy (0.52 vs 0.52 for the shape-variant comparison).
- The inference-efficient law gives a Spearman correlation of 1.00 for 1B model ranking versus -0.40 for Chinchilla.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| include model aspect ratio in the scaling law | Scaling-law loss prediction · Held-out model losses | MSE: 0.0033 to 0.0006 | Helped |
| include model aspect ratio in the scaling law | Scaling-law loss prediction · Held-out model losses | R²: 0.9895 to 0.9982 | Helped |
| include model aspect ratio in the scaling law | 1B model loss ranking · 1B models | Spearman correlation: -0.40 to 1.00 | Helped |
| use a wider, shallower architecture (3072 hidden size, 12 layers) instead of the Open-LM-1B shape (ablation) | Morph-1B · A100, input 128, output 256 | Latency: 3.61 to 1.96 seconds | Helped |
| exclude over-training data from fitting (ablation) | Inference-Efficient scaling law · Scaling-law loss prediction | MSE: 0.0006 to 0.1165 | Hurt |
| exclude over-training data from fitting (ablation) | Inference-Efficient scaling law · Scaling-law loss prediction | MSE: 0.1165 vs 0.9825 | Helped |
| randomly select model shapes for fitting (ablation) | Inference-Efficient scaling law · Scaling-law loss prediction | MSE: 0.0008 vs 0.0198 | Helped |
| randomly select model shapes for fitting (ablation) | Inference-Efficient scaling law · Scaling-law loss prediction | R²: 0.9973 vs 0.9369 | Helped |
| increase model aspect ratio by making the model wider and shallower (ablation) | Models with fixed parameter count · End-to-end inference | Latency | Helped |
Surprising findings
- Models of the same size can differ up to 3.5x in inference latency depending on architecture.
- Smaller models (e.g., MiniCPM-1B) can have higher latency than larger ones (e.g., Qwen2.5-14B).
Limitations
- Scaling to 7B models was not possible due to resource limits.
- Only a small set of model sizes (up to 1.5B trained) and one hardware/setting for latency; results on other hardware (A30, vLLM) are in appendix only.
Experimental setup
Datasets: DCLM-Baseline
Benchmarks: ARC-Easy ARC-Challenge BoolQ COPA HellaSwag LAMBADA PIQA WinoGrande MMLU Jeopardy Winograd
Metrics: accuracy inference latency (s) MSE R^2 relative prediction error Spearman rank correlation
Baselines: Chinchilla scaling law Open-LM-1B OPT-1.3B Pythia-1.3B OPT-IML-1.3B GPT-NeoX-1.3B
Builds on: Chinchilla scaling law Open-LM DCLM-Baseline
Closest papers by meaning
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.95)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.92)
- Parallel Scaling Law for Language Models (NeurIPS 2025 · similarity 0.92)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.92)
- Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (ICML 2024 · similarity 0.92)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.