Research map / Large language models / AI and society
Position: Enough of Scaling LLMs! Lets Focus on Downscaling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
A position paper arguing for "downscaling" LLMs: it derives that CO2 emissions scale linearly with parameters and data, and proposes a pipeline (data pruning, pruning, domain continual pretraining, ensembling) with a theoretical condition under which ensembles of pruned small models beat a base model at equal compute.
Problem
Neural scaling laws drive ever-larger LLMs with exponentially rising computational, environmental, and deployment costs; the paper asks whether downscaling (smaller models trained on curated data, pruned and ensembled) can preserve performance at lower cost.
Approach
Combines a carbon-cost derivation (CO2 linear in N and D), the P2 pruning post-training law, the deep-ensemble scaling law, and the D-CPT law into a proposition giving an n for which an ensemble of pruned models beats the base model at the same compute, and proposes a multi-stage downscaling pipeline.
Key results
- Derives that CO2eq scales linearly with model parameters N and training data D (CO2eq = (K1+K2)·N·D), while loss decreases only logarithmically.
- Using loose fitted constants for LLaMA-3-8B, the proposition suggests an ensemble of 8 or more pruned 1B-parameter versions could outperform the original model at no extra compute.
- Estimates that a 10% performance gain would require about 329% more carbon emissions under the derived relation P ∝ CO2eq^0.08.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use REBASE inference strategies with Llemma-7B instead of Llemma-34B | Llemma · Math tasks | computational cost at comparable performance: at half the computational cost | Helped |
| Prune more than 30% of model weights | Language models including OPT and LLaMA · Pruning | fact recall: more than 30% | Hurt |
| Prune up to 60-70% of model weights | Language models including OPT and LLaMA · Pruning | in-context learning: up to 60-70% | No clear effect |
Surprising findings
- Fact recall degrades after about 30% pruning while in-context learning tolerates 60-70% pruning, suggesting capability-dependent downscaling.
- Ensembling of pruned smaller models is argued to match or beat a single large model without additional compute.
Limitations
- Ensemble proposition relies on loose assumptions and constants fitted in other settings (e.g., CNN ensembles), without empirical validation of the pipeline.
- Downscaled SLMs struggle with complex reasoning and multi-field expertise, and ensembles add latency and resource use at inference.
Experimental setup
Metrics: estimated CO2eq (LLMCarbon) theoretical ensemble loss condition
Baselines: LLaMA-3-8B
Builds on: Kaplan scaling law Chinchilla scaling law LLMCarbon P2 law D-CPT law
Closest papers by meaning
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.93)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.90)
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.90)
- Selecting Large Language Model to Fine-tune via Rectified Scaling Law (ICML 2024 · similarity 0.89)
- Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies (NeurIPS 2024 · similarity 0.89)
- Strong Model Collapse (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.