AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition

Weidong Zhou, Fengze Liu, Binbin Liu, Ping Guo, Zijun Wang, Bingni Zhang, Yifan Zhang, Yifeng Yu, Xiaohuan Zhou, Taifeng Wang

ICML 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
select top 20% FineWebEdu data instead of random data1.2B model, 30B tokens · average of four benchmarksaccuracy (%): 31.46% -> 35.61%Improved
select top 5% FineWebEdu data instead of random data1.2B model, 30B tokens · average of four benchmarksaccuracy (%): 31.46% -> 36.52%Improved
tighten selection from top 20% to top 5% of FineWebEdu1.2B model, 30B tokens · average of four benchmarksaccuracy (%): 35.61% -> 36.52%Improved
tighten selection from top 20% to top 5% of FineWebEdu1.2B model, 30B tokens · HellaSwag and TriviaQAaccuracy (%): 55.26% -> 55.14% (HellaSwag); 20.05% -> 19.25% (TriviaQA)Hurt
tighten selection from top 20% to top 5% of FineWebEdu1.2B model, 30B tokens · ARC-C and MMLU-LightEvalaccuracy (%): 34.30% -> 37.20% (ARC-C); 32.82% -> 34.50% (MMLU)Improved
replace log(K) normalization with constant normalization in InfoLaw (ablation)InfoLaw · large models, larger token budgetsloss predictionHurt
replace log(K) normalization with power-law normalization in InfoLaw (ablation)InfoLaw · Information vs validation lossfit qualityHurt
use InfoLaw instead of traditional power-law scaling in loss-compute planescaling law · extrapolation to 7B models fit on 252M-1.2Bloss predictionImproved
use logarithmic lambda(N) instead of exponential or power-law forms (ablation)InfoLaw · lambda vs N fitfit and extrapolationImproved
raise overtrain degree from m=3.6 to m'=25 with same fitted f and lambdaInfoLaw · 1.2B model, 640B tokensloss-info curveNo clear effect
increase repetition via HQ recipe instead of MQ (top bucket about 16x vs 10x)850M model · final evaluation lossvalidation lossHurt
use InfoLaw-searched optimal recipe instead of preset recipes2.5B model · validation lossvalidation lossImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy