Research map / Architectures and efficiency / RNNs and normalization
Mechanistic Design and Scaling of Hybrid Architectures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mechanistic Architecture Design (MAD), a pipeline of small synthetic token-manipulation tasks that predict compute-optimal scaling, and uses it to find hybrid (striped) architectures that outperform Transformer++, Hyena and Mamba at scale.
Problem
Architecture design is costly and opaque; it is unclear whether cheap small-scale synthetic capability tests can predict scaling-law performance of new architectures, and how to best combine computational primitives (attention, Hyena, Mamba, MoE) into hybrids.
Approach
Use a suite of synthetic token-manipulation unit tests (in-context recall, fuzzy/noisy recall, selective copying, compression, memorization) at small scale with iso-state normalization to shortlist architectures, then validate with IsoFLOP compute-optimal and state-optimal scaling-law analysis on The Pile; striped hybrids of attention with recurrent/convolutional primitives and sparse MoE channel
Key results
- Striped architectures outperform non-striped ones on MAD composites by an average 8.1% accuracy.
- Aggregate MAD scores are linearly correlated with compute-optimal perplexity on The Pile across compute budgets.
- New hybrid architectures reduce perplexity by up to 20% at the same compute budget versus Transformer++, Hyena and Mamba.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Interleave different computational primitives in a striped topology | Model architectures evaluated on MAD · MAD synthetic tasks | accuracy: +8.1% accuracy | Helped |
| Expand total state dimension through heads | Sequence-mixing architectures · MAD synthetic tasks | accuracy: +2.3% accuracy | Helped |
| Replace SwiGLU channel mixers with sparsely activated mixture-of-experts channel mixers | Model architectures evaluated on MAD · MAD synthetic tasks | accuracy: +1.7% accuracy | Helped |
| Use hybrid architectures based on hybridization and sparsity | Language models evaluated on The Pile · The Pile, compute-optimal scaling | perplexity: up to $20\%$ reduction in perplexity | Helped |
| Use a 25% attention hybridization ratio in striped models | StripedHyena · The Pile, compute-optimal scaling | perplexity | Helped |
| Use a 10% hybridization ratio when optimizing for state | Striped hybrid architectures · The Pile, state-optimal scaling | perplexity-state trade-off | Helped |
| Train smaller models for longer outside the compute-optimal frontier | Hybrid architectures · The Pile, overtrained regime | perplexity gap | Helped |
| Evaluate at byte resolution rather than sub-word tokenization | Transformer++, Mamba, Hyena, and StripedHyena · Byte-resolution compute-optimal scaling | perplexity | Mixed |
Surprising findings
- Compute-optimal hybridization ratio of about 25% attention is optimal, while state-optimal accounting shifts it to 10%.
- Byte-level results show attention models yield significantly higher perplexity, so model rankings vary across tokenization.
Limitations
- MAD extrapolates from small two-block models and has not been applied to deeper or more complex hybrid topologies.
- Correlation across different architecture classes is noisier than within a class.
- Memorization task showed little variation across architectures, so it did not correlate with scale performance.
Experimental setup
Datasets: The Pile MAD synthetic tasks
Benchmarks: MAD synthetic tasks The Pile
Metrics: accuracy perplexity evaluation loss rank correlation
Baselines: Transformer++ Hyena Mamba Multi-Head Hyena Gated Linear Attention
Builds on: Chinchilla scaling laws Hyena Mamba Transformer++ gated linear attention
Closest papers by meaning
- STAR: Synthesis of Tailored Architectures (ICLR 2025 · similarity 0.87)
- Memory Layers at Scale (ICML 2025 · similarity 0.86)
- Principled Architecture-aware Scaling of Hyperparameters (ICLR 2024 · similarity 0.85)
- Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers (NeurIPS 2025 · similarity 0.85)
- Learning Rate Transfer for Hybrid Transformer-SSM Architectures (NeurIPS 2026 · similarity 0.85)
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.