Research map / Large language models / AI and society
Position: AI Scaling: From Up to Down and Out
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper proposing a holistic AI scaling framework of Scaling Up, Scaling Down, and Scaling Out, arguing future progress lies in efficiency-oriented downscaling and ecosystem-oriented scaling out.
Problem
Scaling Up of models faces data, compute, energy and cost bottlenecks, while the field needs efficiency, adaptability and collaboration across diverse applications.
Approach
Reframes AI scaling as three complementary paradigms: Scaling Up builds foundation models, Scaling Down distills them into efficient core models via pruning, quantization, distillation and MoE, and Scaling Out derives specialized interfaces and a distributed ecosystem from the core model.
Key results
- Position paper with no new experiments; argues Scaling Up faces data, compute and energy bottlenecks and that Scaling Down and Scaling Out are the future trajectory.
- Cites example sparsity figures such as DeepSeek-V3 activating 37B of 671B parameters per token (5.5%) and Mixtral 2 of 8 experts per token.
Limitations
- No empirical evaluation of the proposed framework; claims are argumentative and illustrative.
- Focus centered on LLMs, with other modalities only briefly addressed.
- Scaling Out and Scaling Down metrics are proposed without validation.
Experimental setup
Builds on: LoRA GPTQ AWQ Wanda speculative decoding Mixture of Experts Retrieval-Augmented Generation Toolformer
Closest papers by meaning
- AI Progress Should Be Measured by Capability-Per-Resource, Not Scale Alone: A Framework for Gradient-Guided Resource Allocation in LLMs (NeurIPS 2025 · similarity 0.87)
- Balancing Humans and Machines: A Study on Integration Scale and Its Impact on Collaborative Performance (AAAI 2024 · similarity 0.85)
- A Tale of Tails: Model Collapse as a Change of Scaling Laws (ICML 2024 · similarity 0.84)
- Resource Democratization: Is Compute the Binding Constraint on AI Research? (AAAI 2024 · similarity 0.84)
- Position: Levels of AGI for Operationalizing Progress on the Path to AGI (ICML 2024 · similarity 0.84)
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.