Research map / Architectures and efficiency / Neural architecture search
STAR: Synthesis of Tailored Architectures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STAR, an evolutionary architecture-synthesis framework that encodes hierarchical linear input-varying (LIV) operators as integer genomes and evolves backbones for language-model quality, size, and cache efficiency.
Problem
Automated architecture search for language models has struggled to produce well-conditioned, comprehensive search spaces that yield improvements in quality and efficiency over standard Transformers and striped hybrids.
Approach
Grounds the search space in linear input-varying systems, encoding featurization, operator structure, and backbone composition (including featurizer and feature-group sharing) into a hierarchical integer genome that is optimized with gradient-free evolutionary algorithms (NSGA-II, GA, Firefly) under constraint-preserving mutations.
Key results
- Quality-optimized STAR backbones at 125M outperform Transformer++ and StripedMamba on downstream average (51.0 vs 48.8 and 49.5) with 79M params.
- Quality-and-cache STAR-1B reaches 86MB cache at 4K vs 805MB for Transformer++ (about 90% smaller) and 136MB for StripedMamba, with matched 59.0 average accuracy vs StripedMamba.
- Quality-and-size optimization yields up to 13% parameter reduction (70M vs 80-85M baselines) while outperforming Transformer++ on most downstream tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| evolve architectures for quality rather than use Transformer++ | 125M language models · RedPajama eval | perplexity: 7.3 -> 7.0 | Helped |
| evolve architectures for quality rather than use StripedMamba | 125M language models · LM-Eval-Harness average | accuracy average: 49.5 -> 51.0 | Helped |
| optimize for quality and size | STAR · RedPajama eval | perplexity: 7.2 | No clear effect |
| optimize for quality and size | STAR · parameter count | trainable parameter count: 10% | Helped |
| optimize for quality and cache size | STAR · inference cache at sequence length 4096 | cache size: 40% | Helped |
| scale a synthesized quality-and-cache backbone from 125M to 1B parameters | STAR backbone · 1B RedPajama eval | perplexity: 5.7 | No clear effect |
| scale a synthesized quality-and-cache backbone from 125M to 1B parameters | STAR backbone · 1B model, sequence length 4096 | cache size: 90% and 37% | Helped |
| use full-width and full-depth synthesis instead of motif synthesis (ablation) | STAR synthesis protocol · language modeling | downstream performance | Helped |
| use reduced-width synthesis instead of full-width and full-depth synthesis (ablation) | STAR synthesis protocol · language modeling | successful candidate count | Hurt |
| use NSGA-2 instead of Firefly Algorithm (ablation) | STAR evolutionary optimization · quality and parameter count optimization | parameter count | Helped |
| use Genetic Algorithm instead of NSGA-2 (ablation) | STAR evolutionary optimization · quality and parameter count optimization | predictive quality | Helped |
| use NSGA-2 instead of Genetic Algorithm (ablation) | STAR evolutionary optimization · quality and parameter count optimization | solution diversity | Helped |
Surprising findings
- Synthesis at reduced depth (motifs) or reduced width underperforms full-scale synthesis, so smaller-scale search does not transfer well.
Limitations
- Fixed-depth and fixed-width genomes; variable-depth search not supported.
- LIV search space does not cover all function classes that could be embedded in a backbone.
- Smaller-scale (motif or reduced width) synthesis yields suboptimal results.
Experimental setup
Datasets: RedPajama
Benchmarks: RedPajama perplexity HellaSwag ARC-Easy ARC-Challenge Winogrande PiQA SciQ
Metrics: perplexity accuracy accuracy normalized parameter count KV/state cache size in bytes
Baselines: Transformer++ StripedMamba Firefly Algorithm Genetic Algorithm
Builds on: NSGA-II Mamba Differential Transformer Transformer++ StripedMamba
Closest papers by meaning
- Mechanistic Design and Scaling of Hybrid Architectures (ICML 2024 · similarity 0.87)
- Towards Neural Architecture Search through Hierarchical Generative Modeling (ICML 2024 · similarity 0.85)
- Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers (NeurIPS 2025 · similarity 0.85)
- Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search (NeurIPS 2026 · similarity 0.85)
- einspace: Searching for Neural Architectures from Fundamental Operations (NeurIPS 2024 · similarity 0.84)
- Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.