Research map / Architectures and efficiency / RNNs and normalization
Parallel Scaling Law for Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ParScale, which runs P learnable prefix-transformed copies of the input through the same model in parallel and dynamically aggregates outputs, showing a parallel scaling law where P streams roughly equal O(log P) parameter scaling with better inference memory/latency efficiency.
Problem
Scaling language models usually costs substantial memory (parameter scaling) or latency (inference-time scaling via reasoning tokens); the paper asks whether parallel computation with fixed parameters can provide an efficient, universal scaling axis and how it trades off against parameter scaling.
Approach
Reuse the same parameters across P parallel streams, each distinguished by learnable prefix KV-cache transformations of the input, and aggregate their outputs with a learned, dynamic softmax-weighted sum; theory and fitted scaling laws show P streams act like scaling parameters by about O(log P).
Key results
- Scaling to P=8 on a 1.6B model matches the performance of a 4.4B parameter-scaled model on code, with 22x less memory increase and 6x less latency increase at batch size 1.
- Fitted logarithmic parallel scaling law reaches R^2 up to 0.998 on Stack-V2 and Pile, with k=0.39 for Stack-V2 vs 0.33 for Pile.
- With 1T-token two-stage training, P=8 improves GSM8K from 28.7 to 38.4 (about 34% relative) and MBPP from 51.6 to 56.3 over the P=1 baseline for 1.8B models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase parallel streams from P=1 to P=8 | 1.8B model trained with the two-stage strategy · GSM8K | accuracy: 28.7 -> 38.4 | Helped |
| Increase parallel streams from P=1 to P=8 | 1.8B model trained with the two-stage strategy · MMLU | accuracy: 28.5 -> 35.1 | Helped |
| Increase parallel streams from P=1 to P=8 | Stack-V2-Python pretrained models · HumanEval(+)/MBPP(+) average, 4.4B model | average performance (%): 39.2 -> 45.4 | Helped |
| Increase parallel streams from P=1 to P=8 | Pile pretrained models · Six general lm-evaluation-harness tasks, 4.4B model | average performance (%): 57.2 -> 59.6 | Helped |
| Compare parallel scaling with parameter scaling at matched performance | 1.6B model at batch size 1 · Inference | memory increase: 22× less memory increase | Helped |
| Compare parallel scaling with parameter scaling at matched performance | 1.6B model at batch size 1 · Inference | latency increase: 6× less latency increase | Helped |
| Set label smoothing ε=0.0 rather than ε=0.1 (ablation) | P=2 dynamic weighted sum with Prefix (48 tokens) · Stack-V2-Python, 0.5B model | loss: 1.1276 -> 1.1284 | Hurt |
| Replace dynamic weighted sum with average aggregation (ablation) | P=2 ParScale with Prefix (48 tokens) · Stack-V2-Python, 0.5B model | loss: 1.1276 -> 1.1288 | Hurt |
| Replace dynamic weighted sum with a linear layer (ablation) | P=2 ParScale with Prefix (48 tokens) · Stack-V2-Python, 0.5B model | loss: 1.1276 -> 1.1323 | Hurt |
| Increase prefix length from 48 to 96 tokens (ablation) | P=2 ParScale with dynamic weighted sum (ε=0.1) · Stack-V2-Python, 0.5B model | loss: 1.1276 -> 1.1278 | No clear effect |
| Replace prefix tuning with LoRA (r=2) (ablation) | P=2 ParScale with dynamic weighted sum (ε=0.1) · Stack-V2-Python, 0.5B model | loss: 1.1276 -> 1.1281 | No clear effect |
| Add LoRA (r=2) to Prefix (48 tokens) (ablation) | P=2 ParScale with dynamic weighted sum (ε=0.1) · Stack-V2-Python, 0.5B model | loss: 1.1276 -> 1.1263 | Helped |
Surprising findings
- Varying the input transformation (prefix vs LoRA vs BitFit) and aggregation method changes loss by only around 0.1-0.2%, so the number of streams P is the key factor.
- Parallel scaling reduces the degradation when overfitting on repeated OpenWebText data, with a 3B two-way model matching a 5B parameter model's validation loss at the overfitting point.
Limitations
- The theoretical diversity term is not modeled analytically, so the scaling law is fit empirically with a logarithmic form.
- ParScale multiplies training FLOPs by about P, making from-scratch training expensive, which motivates the two-stage workaround.
- Experiments are limited to P up to 8 and models up to about 4.4B from scratch; behavior for larger P is an open question.
Experimental setup
Datasets: Stack-V2 (Python) Pile FineWeb-Edu Cosmopedia 2 FineMath Stack-Python-Edu OpenWebText SmolTalk
Benchmarks: HumanEval HumanEval+ MBPP MBPP+ MMLU WinoGrande HellaSwag OBQA PiQA ARC GSM8K Minerva Math
Metrics: loss accuracy pass@1 pass@10 memory latency
Baselines: Parameter scaling (dense) Beam search SmolLM-1.7B Llama-3.2-1B Qwen2.5-1.5B gemma-3-1B SmolLM2-1.7B
Builds on: Classifier-free guidance Chinchilla scaling law prefix tuning Qwen-2.5
Closest papers by meaning
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.92)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.92)
- OptScale: Probabilistic Optimality for Inference-time Scaling (AAAI 2026 · similarity 0.91)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.91)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.90)
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.