Research map / Architectures and efficiency / RNNs and normalization
Layer-wise analysis of language models
Probes hidden representations and layer dynamics in pretrained language models.
145 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in RNNs and normalization, keyword cluster “layer · activation · functions”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
22 papers in 2023–24 and 104 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 3.4% to 11.1%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Social LSTM: Human Trajectory Prediction in Crowded Spaces (CVPR 2016 · 3,627 citations)
- 2017 · Diet Networks: Thin Parameters for Fat Genomics (ICLR 2017 · 10 citations)
- 2018 · Multi-Task Zipping via Layer-wise Neuron Sharing (NeurIPS 2018 · 32 citations)
- 2019 · Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains (NeurIPS 2019 · 4 citations)
- 2020 · Identifying Statistical Bias in Dataset Replication (ICML 2020 · 7 citations)
- 2021 · Maximum Roaming Multi-Task Learning (AAAI 2021 · 19 citations)
- 2022 · Locating and Editing Factual Associations in GPT (NeurIPS 2022 · 173 citations)
- 2023 · Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models (NeurIPS 2023 · 24 citations)
- 2024 · Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models (ICML 2024 · 55 citations)
- 2025 · Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (ICLR 2025 · 4 citations)
- 2026 · Resting Neurons, Active Insights: Robustifying Activation Sparsity in LLMs via Spontaneity (ICML 2026 · 0 citations)
Earliest papers here
- Social LSTM: Human Trajectory Prediction in Crowded Spaces (CVPR 2016 · 3,627 citations)
- A Neural Autoregressive Approach to Collaborative Filtering (ICML 2016 · 165 citations)
- Diet Networks: Thin Parameters for Fat Genomics (ICLR 2017 · 10 citations)
What has been tried
68 papers here report 687 measured changes (a component added, removed or swapped, compared with a baseline): helped 61%, no clear effect 13%, hurt 20%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| transfer deeper PLM layers | SS3 (CB513, TS115, CASP12) · accuracy | Helped | Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models (ICML 2024 · 55 citations) | |
| balance sampling by operand digit length | 3-digit addition · overall accuracy | 97.74 -> 98.13 | Helped | Teaching Arithmetic to Small Transformers (ICLR 2024 · 7 citations) |
| Further fine-tune after Shift feature ablation | Bias in Bios balanced set · Profession accuracy | 88.5 -> 93.1 | Helped | Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (ICLR 2025 · 4 citations) |
| add qk-layernorm | LR sweep 3e-4 to 3e-1 · LR sensitivity / training stability | Helped | Small-scale proxies for large-scale Transformer training instabilities (ICLR 2024 · 4 citations) | |
| Use small initialization rather than large initialization | Synthetic anchor functions · Inferential-mapping accuracy | Helped | Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing (NeurIPS 2024 · 3 citations) | |
| use squared-gradient Hessian approximation instead of identity Hessian | sentiment task addition (Avg. accuracy) · accuracy | 94.3 -> 94.5 | Helped | Model Merging by Uncertainty-Based Gradient Matching (ICLR 2024 · 2 citations) |
| Concatenate residual-stream activations across layers | RRXOR · belief-state geometry representation | Helped | Transformers Represent Belief State Geometry in their Residual Stream (NeurIPS 2024 · 2 citations) | |
| add merged linear weights | Eight vision tasks (Avg Acc) · accuracy | 49.8 -> 82.7 | Helped | Towards Minimizing Feature Drift in Model Merging: Layer-wise Task Vector Fusion for Adaptive Knowledge Integration (NeurIPS 2025 · 1 citation) |
| Predict question tokens during training instead of excluding them | DeepMind Math, Mul/Div Multiple Longer · accuracy | 63% -> 36% | Hurt | Do Language Models Use Their Depth Efficiently? (NeurIPS 2025 · 1 citation) |
| add the equivariant decoder to the TabPFN backbone | TabZilla benchmark tasks (ablation evaluation) · % Error reduction over TabPFN | +0.94% | Helped | EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Network (NeurIPS 2025 · 1 citation) |
| Use the proposed fixed PCA-based steering plane instead of Span(h, feature) | Steered generations across tested models and steering angles · perplexity | 5.6141 -> 2.9303 (Qwen2.5-3B-Instruct mean) | Helped | Angular Steering: Behavior Control via Rotation in Activation Space (NeurIPS 2025 · 1 citation) |
| Use full-rank rather than reduced-rank experts | Sparse layer approximation, K=32 · downstream cross-entropy loss | Helped | Towards Interpretability Without Sacrifice: Faithful Dense Layer Decomposition with Mixture of Decoders (NeurIPS 2025 · 1 citation) |
What reviewers will expect
Datasets and baselines named in the 57 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MMLU · 3 of 57 papers
- GSM8K · 3 of 57 papers
- CIFAR100 · 3 of 57 papers
- ImageNet-1k · 3 of 57 papers
- WikiText · 3 of 57 papers
- Alpaca · 3 of 57 papers
- The Pile · 3 of 57 papers
- C4 · 2 of 57 papers
Baselines
- Random · 3 of 57 papers
- Mamba-2 · 2 of 57 papers
- Post-LN · 2 of 57 papers
- DeepNorm · 2 of 57 papers
- Pre-LN · 2 of 57 papers
- Sandwich-LN · 2 of 57 papers
- LayerNorm Scaling · 2 of 57 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 126 papers; from OpenAlex and the papers' affiliations).
- Google · 6 papers
- Mila · 5 papers
- Stanford University · 5 papers
- Google DeepMind · 4 papers
- Shanghai Jiao Tong University · 4 papers
- National University of Singapore · 4 papers
- New York University · 4 papers
- Apple · 4 papers
- NVIDIA · 3 papers
- Microsoft · 3 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Theory of depth and width (223 papers)
- Representation geometry and collapse (169 papers)
- Symmetry and equivariance in loss landscapes (95 papers)
- Mechanistic circuit interpretability (93 papers)
- Sparse dictionary representations (98 papers)
- Width and depth of ReLU networks (53 papers)
Elsewhere in the topic
- Efficient network compression and training (229 papers)
- Dataset distillation and data-efficient training (158 papers)
- Neural scaling laws (155 papers)
- Distributed LLM inference systems (147 papers)
- Dataset condensation and flow estimation (126 papers)
- Differentiable optimization layers (125 papers)
- Scalable approximation and equivariant compute (120 papers)
- Non-autoregressive translation (102 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).