Research map / Architectures and efficiency / Attention and transformers
Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses controlled synthetic pretraining tasks (Depo, Brevo, Capo, Mano, Lano) to compare architectures and introduces Canon layers, lightweight causal conv1d residual mixing across neighboring tokens, which substantially improve Transformers, NoPE, GLA, and Mamba2 on reasoning and knowledge tasks.
Problem
Architectural comparisons at academic pretraining scale (1.3B params, 100B tokens) are dominated by noise, and standard perplexity fails to reveal reasoning differences between architectures.
Approach
Canon layers add learned, position-specific horizontal mixing over a short causal window (kernel size 4, with residual connection) at multiple sublayer positions (A/B/C/D) in any sequence architecture; evaluated via controlled synthetic pretraining tasks that isolate reasoning depth, breadth, knowledge capacity/manipulation, and hierarchical language structure.
Key results
- Canon layers improve RoPE reasoning depth by 2-4x on Depo with a ~0.5% increase in trainable parameters, and lengthen Mano manipulable length by about 30%.
- NoPE+Canon matches or surpasses RoPE+Canon on Depo, whereas NoPE without Canon achieves near-zero performance on all measures.
- Canon-AbCD lifts GLA from 1-hop to 4-hop reasoning depth and makes it competitive with or better than Mamba2 on Brevo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Canon-ABCD | Llama(RoPE) · Depo | reasoning depth: 2-4× | Helped |
| Add Canon-ABCD | Llama(RoPE) · Brevo | reasoning breadth: 30% | Helped |
| Add Canon-ABCD | Llama(RoPE) · Capo | knowledge capacity: 10--15% | Helped |
| Add Canon-ABCD | Llama(RoPE) · Mano | manipulable length: 30% | Helped |
| Add Canon layers | GLA · Depo | reasoning depth: from 1-hop to 4-hop | Helped |
| Add Canon layers | GLA · Brevo | reasoning breadth: doubling | Helped |
| Add Canon layers | GLA · Mano | manipulable length: more than doubling | Helped |
| Apply Canon only in the first block (ablation) | Llama(RoPE) · Depo | reasoning depth | Hurt |
| Remove residual connections from Canon (ablation) | Llama(RoPE) · Transformer ablations | training stability | Hurt |
| Add SiLU after Canon (ablation) | Llama(RoPE) · Transformer ablations | performance | No clear effect |
| Use Canon-ACD outside the attention mechanism (ablation) | GLA · GLA ablations | performance | Helped |
| Add the 1+elu(x) feature map (ablation) | GLA+Canon · GLA ablations | performance | Hurt |
Surprising findings
- Simple fixed random-weight Canon layers already notably improve performance.
- Adding nonlinear activations such as SiLU after Canon layers gives no measurable benefit.
Limitations
- Canon layers alone cannot fully solve very hard long-sequence hierarchical tasks such as cfg3k in Lano.
- Results at academic scale are noisy; real-world gains are reported on a limited set of tasks.
- Dynamic or input-dependent Canon variants were not studied.
Experimental setup
Datasets: SlimPajama FineWeb-Edu cfg3f bioS(N)
Benchmarks: Depo Brevo Capo Mano Lano Babilong 1-hop-L 2-hop-L
Metrics: accuracy bits per parameter KL divergence
Baselines: Llama(RoPE) Llama(NoPE) GLA Mamba2 GDN ALiBi H-Alibi Primer
Builds on: Gated Linear Attention Mamba2 Gated DeltaNet H3 shift-SSM Primer
Closest papers by meaning
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.88)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.88)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.88)
- Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models (ICML 2026 · similarity 0.88)
- Revealing Behavioral Plasticity in Large Language Models: A Token-Conditional Perspective (ICML 2026 · similarity 0.88)
- Memory Layers at Scale (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.