Research map / Architectures and efficiency / Neural architecture search
Composer: A Search Framework for Hybrid Neural Architecture Design
Bilge Acun, Prasoon Sinha, Newsha Ardalani, Sangmin Bae, Alicia Golden, Meghana Madhyastha, Chien-Yu Lin, Fei Sun, Neeraja J. Yadwadkar, Carole-Jean Wu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Composer-discovered hybrid architecture instead of Llama 3.2 | 1B LLM · DCLM validation | validation loss: 2.80 -> 2.77 | Improved |
| use Composite instead of Llama 3.2 | 1B LLM · 6 downstream tasks | average accuracy: 58.69 -> 60.83 | Improved |
| use 1:8 Striped Attention (fewer attention layers, no search) | 1B LLM · DCLM validation | validation loss: 2.80 -> 2.85 | Hurt |
| scale down width during search (ablation) | Composer 6-layer search · 1B DCLM validation | validation loss and search cost: loss -0.02 to -0.04; cost 6.38x lower | Improved |
| use one-shot or middle-incremental search instead of end-incremental search | Composer Search Engine · 1B DCLM validation | validation loss: 0.01-0.02 lower loss | Improved |
| stretch rather than stack discovered architecture when search depth >= 16 layers (ablation) | Composer Extrapolator · 1B DCLM validation | validation loss | Improved |
| stretch rather than stack from very small searches (e.g. 2A+4M) (ablation) | Composer Extrapolator · 1B DCLM validation | validation loss | Hurt |
| search over more than 16 layers (ablation) | Composer Search Engine · 1B DCLM validation | validation loss | Hurt |
| use MAD synthetic dataset instead of Large-Scale DCLM for evaluator (ablation) | Composer Evaluator · search cost | search cost: over 8x lower cost | Improved |
| use Chinchilla-scaled small DCLM (4M model) for search (ablation) | Composer Evaluator · 1B DCLM validation | validation loss | No clear effect |
| use 1:2 attention-to-MLP ratio instead of 1:1 (ablation) | hybrid LLM · 1B DCLM validation | validation loss | Improved |
| use Composite instead of Llama 3.2 | 1B LLM · Training and inference efficiency | throughput/latency: 1.25x throughput, 1.33x latency, 1.69x KV cache | Improved |
Closest papers by meaning
- Mechanistic Design and Scaling of Hybrid Architectures (ICML 2024 · similarity 0.87)
- RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy (ICML 2025 · similarity 0.86)
- Zebra-Llama: Towards Extremely Efficient Hybrid Models (NeurIPS 2025 · similarity 0.85)
- NADER: Neural Architecture Design via Multi-Agent Collaboration (CVPR 2025 · similarity 0.85)
- STAR: Synthesis of Tailored Architectures (ICLR 2025 · similarity 0.85)
- The Mamba in the Llama: Distilling and Accelerating Hybrid Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
