AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

When Does Sparsity Mitigate the Curse of Depth in LLMs

Dilxat Muhtar, Xinyuan Song, Sebastian Pokutta, Max Zimmer, Nico Pelleriti, Thomas Hofmann, Shiwei Liu

ICML 2026 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add moderate weight decay (0 to 0.1) (ablation)1.2B Pre-LN LLM · validation perplexityperplexity: 15.63 -> 14.83Improved
raise weight decay to 3.0 (ablation)1.2B Pre-LN LLM · validation perplexityperplexity: 14.83 -> 773.42Hurt
increase training sequence length from 256 to 2048 (ablation)1.2B Pre-LN LLM · validation perplexityperplexity: 18.51 -> 14.51Improved
increase training sequence length to 8192 (ablation)1.2B Pre-LN LLM · validation perplexityperplexity: 14.52 -> 16.30Hurt
use multi-query attention (G=16) instead of MHA (ablation)1.2B Pre-LN LLM · validation perplexityperplexity: 14.52 -> 14.47Improved
replace dense FFN with fine-grained MoE (1B active) (ablation)Dense-1B · validation perplexityperplexity: 14.52 -> 13.82Improved
replace dense FFN with fine-grained MoE (400M active) (ablation)Dense-400M · validation perplexityperplexity: 17.56 -> 15.89Improved
naively increase depth from L=16 to L=32 at constant parameters1.2B Pre-LN LLM · ARC-C and HellaSwag averageaccuracy and usefulness score: usefulness 0.75 -> 0.53; accuracy drops by 0.5Hurt
add GQA (G=2) and MoE on top of depth L=32L=32 1.2B model · ARC-C and HellaSwag averageaverage accuracy: to 44.1 (usefulness 0.75)Improved
add weight decay 1.0 on L=32 model (ablation)L=32 1.2B model · ARC-C and HellaSwag averageaccuracy and usefulnessHurt
scaled initialization instead of normal initialization (ablation)L=16 model, T=1024 · lossloss: 2.70 -> 2.71No clear effect
add longer sequence length T=4096 on top of scaled initialization (ablation)L=16 model · lossloss: 2.71 -> 2.64Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy