AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Inverse Depth Scaling From Most Layers Being Similar

Yizhou Liu, Sara Kangaslahti, Ziming Liu, Jeff Gore

ICML 2026 · Large language models › Language, safety and interpretability · approach: Linguistic encoding in LLM representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase LLM depth (fit of decomposed scaling law)LLMs (Chinchilla-reconstructed data) · LLM pretraining lossdepth-dependent loss exponent: alpha_l = 1.2 ± 0.3Improved
increase depth in LLMs (Pythia family)Pythia models · FineWeb hidden-state updatesmean update angle in middle layersImproved
tie teacher weights across layers (smooth dynamics) at low teacher temperature (ablation)toy residual student · teacher-student toy model, depth 6 to 48depth exponent alpha_l: alpha_l about 1 (imperfect training)No clear effect
tie teacher weights and raise teacher temperature (ablation)toy residual student · teacher-student toy modeldepth exponent alpha_l: 1 -> 3Improved
train longer on tied-teacher toy model at temperature 1 (ablation)toy residual student · teacher-student toy modeldepth exponent alpha_l: near 1 early -> approaches 3 lateImproved
use independent teacher weights across layers (non-smooth dynamics) (ablation)toy residual student · teacher-student toy modeldepth exponent alpha_l: alpha_l about 1 across temperaturesNo clear effect
mimic higher-order integration scheme in the student architecture (tied teacher) (ablation)toy residual student · teacher-student toy modeldepth exponent alpha_lImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy