Research map / Architectures and efficiency / RNNs and normalization
Do Language Models Use Their Depth Efficiently?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes residual stream, layer skipping, residual erasure, and cross-model linear maps in Llama 3.1, Qwen 3/2.5, and OLMo 2 to show deeper LLMs spread the same computations over more layers rather than composing new higher-order computations.
Problem
Whether modern deep LLMs use their depth efficiently, i.e., compose features into higher-order computations unavailable to shallow models, or merely spread the same computation over more layers.
Approach
Combines residual-stream contribution analysis, causal layer-skipping and residual-erasure interventions, logit-lens probing, and linear maps between independently trained shallow and deep models to test whether later layers perform new computation; finds second-half layers mostly refine the output distribution and depth does not scale with problem complexity.
Key results
- Layers in the second half of Llama 3.1 70B contribute much less to the residual stream than the first half, with a sharp drop near the middle and a phase transition in cosine similarity.
- Skipping layers in the second half has minimal effect on future-token computations, while logit-lens KL divergence to the final output drops around the same depth.
- Linear maps between Qwen 2.5 1.5B and 14B layers show a clear diagonal pattern, so same-relative-depth layers map best, and depth does not track multihop or MATH difficulty.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model depth | 132 Open LLM base models · Open LLM Leaderboard | performance | Helped |
| Use layers in the second half of the network | Llama 3.1 70B · GSM8K | relative layer contribution | Mixed |
| Skip layers in the second half | Llama 3.1 70B · GSM8K future-token predictions | effect on future predictions | No clear effect |
| Increase problem difficulty or number of reasoning hops | Llama models · MATH and MQuAKE | depth score | No clear effect |
| Predict question tokens during training instead of excluding them (ablation) | MoEUT · DeepMind Math, Mul/Div Multiple Longer | accuracy: 63% -> 36% | Hurt |
| Predict question tokens during training instead of excluding them (ablation) | Standard Transformer · DeepMind Math, Mul/Div Multiple Longer | accuracy: 48% -> 41 | Hurt |
| Use MoEUT instead of a standard Transformer | DeepMind Math models trained from scratch · DeepMind Math, Mul/Div Multiple Longer | accuracy: 41 -> 36; 48% -> 63% | Mixed |
| Map activations between layers at the same relative depth | Qwen 2.5 1.5B and Qwen 2.5 14B · Layer activation correspondence | linear-map prediction error | Helped |
| Train MoEUT without predicting question tokens (ablation) | MoEUT · DeepMind Math | depth of answer computation | Helped |
| Fine-tune Llama 3.2 3B on arithmetic (ablation) | Llama 3.2 3B · DeepMind Math arithmetic | computation depth | Mixed |
| Use instruction-tuned weights instead of base weights (ablation) | Llama 3.1 70B · GSM8K future-token predictions | layer importance | Mixed |
| Use Qwen 3 32B instead of smaller Qwen models | Qwen 3 · GSM8K future-token predictions | layer importance | Mixed |
Surprising findings
- Depth-dependent computation does not scale with problem difficulty or number of hops; the models show no evidence of deeper composition for harder problems.
- Fine-tuning Llama 3.2 3B on arithmetic produces only marginal increases in computation depth, suggesting fine-tuning does not fundamentally change depth use.
Limitations
- Case study limited to Llama, Qwen, and OLMo 2 models; findings may not generalize to other architectures.
- Linear map analysis relies on a single model pair (Qwen 2.5 1.5B and 14B) due to compute cost.
- Most interventions use only 10-20 examples and rely on manual inspection of individual trained models.
Experimental setup
Datasets: GSM8K MATH MQuAKE DeepMind Math Open LLM Leaderboard HELM Lite
Benchmarks: GSM8K MATH MQuAKE DeepMind Math Open LLM Leaderboard HELM Lite
Metrics: relative contribution norm cosine similarity relative change in layer contribution output probability change (L2) KL divergence (logit lens) top-5 overlap integrated gradients depth score relative linear map prediction error accuracy
Baselines: MoEUT standard Transformer
Builds on: Logit lens Integrated gradients MoEUT Universal Transformers
Closest papers by meaning
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.92)
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs (ICLR 2026 · similarity 0.89)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.89)
- Exploring Activation Patterns of Parameters in Language Models (AAAI 2025 · similarity 0.88)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.88)
- Skip a Layer or Loop It? Learning Program-of-Layers in LLMs (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.