AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws

Xu Ouyang, Deyi Liu, Yuhang Cai, Jing Liu, Yuan Yang, Chen Zheng, Thomas Hartvigsen, Yiyuan Ma

ICML 2026 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use token-dependent noise term c(DN)^gamma instead of size-only cN^gamma (ablation)Shannon Scaling Law · Pythia 10 dBR^2: 0.8035 -> 0.9555Improved
same, OLMo2 at 10 dB (ablation)Shannon Scaling Law · OLMo2 10 dBR^2: 0.8695 -> 0.8695No clear effect
same, Pythia 40 dB (ablation)Shannon Scaling Law · Pythia 40 dBR^2: 0.9895 -> 0.9895No clear effect
use 9-parameter full law rather than 6-parameter simplified for joint extrapolation (ablation)Shannon · Pythia k=5,j=12pooled R^2: 0.673 -> 0.847Improved
use 6-parameter simplified law for single-axis model extrapolation (ablation)Shannon · Pythia k=4pooled R^2: 0.787 -> 0.837Improved
use Shannon law instead of Asymmetric lawScaling law fit · GPTQ 2-bit PythiaR^2: 0.9499 -> 0.9602Improved
use Shannon law instead of ChinchillaScaling law fit · Gaussian noise Pythia avgR^2: 0.5289 -> 0.9613Improved
use Shannon law instead of OpenAI lawScaling law fit · SFT StarCoder avgR^2: -1.010 -> 0.937Improved
use Shannon law instead of Asymmetric LawScaling law fit · SFT GSM8K avgR^2: 0.896 -> 0.936Improved
use Shannon law instead of Asymmetric LawScaling law fit · Gaussian noise OLMo2 10 dBR^2: 0.8708 -> 0.8695Hurt
use Shannon-Simpl instead of Shannon 9pScaling law extrapolation · Pythia token j=15pooled R^2: 0.941 -> 0.945No clear effect
use Shannon instead of OpenAIScaling law · Pythia model extrapolation k=4pooled R^2: -0.048 -> 0.787Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy