Research map / Architectures and efficiency / Attention and transformers
Transformer Layers as Painters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of frozen pretrained transformers (mainly Llama2-7B and BERT-Large) showing middle layers share a representation space and tolerate skipping, reordering, and parallel execution, with graceful degradation.
Problem
Understanding how the layers of pretrained transformers interact, and whether they can be skipped, reordered, or run in parallel without retraining.
Approach
Treats transformer layers as "painters" on a residual canvas and tests frozen pretrained models under interventions (skipping, repeating, reversing, randomizing, parallelizing, looping middle layers) without any weight updates, measuring graceful vs catastrophic degradation.
Key results
- Llama2-7B tolerates skipping middle layers with graceful degradation, while skipping early layers is catastrophic.
- Middle Repeat (reusing the center layer) is the most damaging variant, quickly degrading to random baseline performance.
- Looped parallel execution of middle layers performs best among variants for both BERT and Llama2-7B, and GSM8K is the benchmark most harmed by parallelization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| skip middle layers (ablation) | Llama2-7B · Llama2 benchmarks | normalized median | Hurt |
| skip middle layers (ablation) | BERT-Large · GLUE | unnormalized average | Hurt |
| replace middle layers with copies of the center layer (ablation) | Llama2-7B · Llama2 benchmarks | normalized median | Hurt |
| replace middle layers with copies of the center layer (ablation) | BERT-Large · GLUE | unnormalized average | Hurt |
| reverse the order of middle layers (ablation) | Llama2-7B · Llama2 benchmarks | normalized performance | Hurt |
| reverse the order of middle layers (ablation) | BERT-Large · GLUE | unnormalized average | Hurt |
| randomize the order of middle layers (ablation) | Llama2-7B · Llama2 benchmarks | normalized performance | Hurt |
| randomize the order of middle layers (ablation) | BERT-Large · GLUE | unnormalized average | Hurt |
| randomize middle-layer order instead of reversing it (ablation) | Llama2-7B · Llama2 benchmarks | normalized performance | Helped |
| run middle layers in parallel (ablation) | Llama2-7B · GSM8K | performance | Hurt |
| run middle layers in parallel (ablation) | Llama2-7B · Llama2 benchmarks | normalized performance | Hurt |
| loop the parallelized layer 3 times instead of once (ablation) | Llama2-7B · Llama2 benchmarks | normalized performance | Helped |
Surprising findings
- Repeating the center layer is far worse than simply skipping the same middle layers.
- Randomly reordering middle layers outperforms reversing their order.
Limitations
- No fine-tuning of the frozen models is used in main experiments, so adaptation to modifications is not explored.
- Explanation for robustness is left to future work; the study is empirical only.
- Results for BERT's GLUE use a fine-tuned head and are partly not directly comparable to LLM benchmarks.
Experimental setup
Datasets: LAMBADA GLUE
Benchmarks: ARC HellaSwag GSM8K WinoGrande LAMBADA GLUE
Metrics: normalized median accuracy unnormalized average score perplexity average cosine similarity
Baselines: Skip layers Middle Repeat Reversed order Random order Parallel layer Looped parallel
Builds on: residual connections ShortGPT ShortenedLlama
Closest papers by meaning
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture (ICML 2025 · similarity 0.91)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.89)
- Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities (NeurIPS 2025 · similarity 0.89)
- Impact of Layer Norm on Memorization and Generalization in Transformers (NeurIPS 2025 · similarity 0.88)
- Simplifying Transformer Blocks (ICLR 2024 · similarity 0.87)
- SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.