Research map / Large language models / AI and society
AI Progress Should Be Measured by Capability-Per-Resource, Not Scale Alone: A Framework for Gradient-Guided Resource Allocation in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing LLM progress should be measured by capability-per-resource, proposing a theoretical gradient-guided partial-update/data-selection framework and "gradient blueprints" released by foundation developers for downstream adapters.
Problem
Scaling-driven AI development has large environmental costs and widens resource inequality, and there is no formal resource-conscious framework for when to stop pretraining and which parameters or data to update during adaptation.
Approach
Argues for a capability-per-resource (delta Psi / delta Gamma) objective and proposes a two-stage paradigm: marginal-return pretraining stopping rules for foundation labs and influence-guided partial-parameter adaptation for downstream users, bridged by gradient blueprints. Theory claims heavy-tailed gradient power laws make updating only high-influence parameters beat full tuning on performance-p
Key results
- Toy power-law example (N=10,000, alpha=1.5) claims top-10% of parameters contribute nearly 50% of gradient norm; no experiments reported.
- Proposition claims partial updates strictly outperform full tuning on performance-per-resource, with proof relying on assumed power-law form and a numerical example (alpha=1, beta=4, gamma=0.6 giving 0.228 vs 0.2).
- Multiplicative example: 80% performance from 20% parameters and 90% from 30% data yields 72% performance at 6% cost (claimed 12x), a hypothetical illustration rather than measured result.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Update only the top-10% of parameters by gradient influence instead of all parameters | LLM fine-tuning under the paper's power-law gradient and resource-cost assumptions · Toy illustration | memory/performance ratio: 5$\times$ | Helped |
| Select high-influence parameters and data examples rather than using all parameters and data | LLM training under the paper's illustrative independence assumptions · Parameter-data selection example | performance-per-resource: 72% performance at only 6% of the resource cost (a 12× boost) | Helped |
| Use gradient norms to select influential parameters instead of computing second-order influence measures | Gradient-guided parameter selection · Transformer models | selection quality | No clear effect |
Surprising findings
- The paper's own appendix numerical example yields a combined efficiency of about 3.17x, inconsistent with the 12x claim in the main text.
- The appendix's stated assumption that resource cost is sequential in parameter count is challenged by an internal review note that autograd is parallelized, so the proposition's cost model may not hold.
Limitations
- No empirical experiments validate the power-law gradient assumption or the partial-update advantage.
- The resource cost model assumes cost linear in trainable parameters, which the paper itself treats as an assumption and which may not hold under parallel autograd.
- Proof of the main proposition shows the inequality only for specific parameter values, not in general.
Experimental setup
Datasets: PubMedQA
Metrics: performance-per-resource ratio (delta Psi / delta Gamma)
Baselines: LoRA QLoRA full-parameter fine-tuning
Builds on: LoRA QLoRA Chinchilla scaling laws
Closest papers by meaning
- Position: AI Scaling: From Up to Down and Out (ICML 2025 · similarity 0.87)
- UltraHorizon: Benchmarking LLM-Agent Capabilities in Ultra Long-Horizon Scenarios (ICML 2026 · similarity 0.86)
- FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics (NeurIPS 2026 · similarity 0.86)
- Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? (ICML 2025 · similarity 0.86)
- Increased Compute Efficiency and the Diffusion of AI Capabilities (AAAI 2025 · similarity 0.86)
- EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.