Research map / Large language models / Language, safety and interpretability
InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition
Weidong Zhou, Fengze Liu, Binbin Liu, Ping Guo, Zijun Wang, Bingni Zhang, Yifan Zhang, Yifeng Yu, Xiaohuan Zhou, Taifeng Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select top 20% FineWebEdu data instead of random data | 1.2B model, 30B tokens · average of four benchmarks | accuracy (%): 31.46% -> 35.61% | Improved |
| select top 5% FineWebEdu data instead of random data | 1.2B model, 30B tokens · average of four benchmarks | accuracy (%): 31.46% -> 36.52% | Improved |
| tighten selection from top 20% to top 5% of FineWebEdu | 1.2B model, 30B tokens · average of four benchmarks | accuracy (%): 35.61% -> 36.52% | Improved |
| tighten selection from top 20% to top 5% of FineWebEdu | 1.2B model, 30B tokens · HellaSwag and TriviaQA | accuracy (%): 55.26% -> 55.14% (HellaSwag); 20.05% -> 19.25% (TriviaQA) | Hurt |
| tighten selection from top 20% to top 5% of FineWebEdu | 1.2B model, 30B tokens · ARC-C and MMLU-LightEval | accuracy (%): 34.30% -> 37.20% (ARC-C); 32.82% -> 34.50% (MMLU) | Improved |
| replace log(K) normalization with constant normalization in InfoLaw (ablation) | InfoLaw · large models, larger token budgets | loss prediction | Hurt |
| replace log(K) normalization with power-law normalization in InfoLaw (ablation) | InfoLaw · Information vs validation loss | fit quality | Hurt |
| use InfoLaw instead of traditional power-law scaling in loss-compute plane | scaling law · extrapolation to 7B models fit on 252M-1.2B | loss prediction | Improved |
| use logarithmic lambda(N) instead of exponential or power-law forms (ablation) | InfoLaw · lambda vs N fit | fit and extrapolation | Improved |
| raise overtrain degree from m=3.6 to m'=25 with same fitted f and lambda | InfoLaw · 1.2B model, 640B tokens | loss-info curve | No clear effect |
| increase repetition via HQ recipe instead of MQ (top bucket about 16x vs 10x) | 850M model · final evaluation loss | validation loss | Hurt |
| use InfoLaw-searched optimal recipe instead of preset recipes | 2.5B model · validation loss | validation loss | Improved |
Closest papers by meaning
- Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance (ICLR 2025 · similarity 0.92)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.91)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.91)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.90)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.90)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
