Research map / Large language models / Agents, code and math
Language Modeling by Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Genesys, a multi-agent LLM system with genetic-programming-style unit-wise code generation and a Ladder-of-Scales budget that discovers and pretrains 1,062 new LM architectures (14M-350M), with some designs competitive with GPT2 and Mamba2.
Problem
Automating the discovery of novel language model architectures that improve on the transformer, under limited compute and with verifiable outcomes, while managing unreliable LLM code generation.
Approach
A multi-agent proposer-reviewer and planner-coder-observer pipeline generates architectures unit-by-unit (Viterbi-style, freezing validated units) within a genetic programming loop over a GAU tree, and verifies designs through a Ladder-of-Scales pretraining budget.
Key results
- Genesys-discovered designs outperform baselines on 6/9 benchmarks at 350M scale and 7/9 at 125M, per the paper's reported tables.
- Unit-based generation raises valid code rate from 6% (Direct) to 92% (Full), with ~86 percentage point improvement in successful design generation.
- Full system reaches End fitness improvement of 4.10% with Sharpe ratio 0.69 over 300 designs, versus 0.567 Sharpe without literature access.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove literature access (ablation) | Genesys · 300-design evolutionary run | fitness improvement (Δ): reduced Δ by 0.73% | Hurt |
| remove literature access (ablation) | Genesys · 300-design evolutionary run | Sharpe Ratio: 0.69 -> 0.567 | Hurt |
| remove experiment verification and fitness information from selection (ablation) | Genesys · 500- and 1000-design evolutionary runs | fitness improvement (Δmax): doubled Δmax | Hurt |
| remove experiment verification and fitness information from selection (ablation) | Genesys · verification process | error rate: ~19% higher error rate | Hurt |
| limit search to the five starting designs, with no acquired discovery designs or knowledge (ablation) | Genesys · 300-design evolutionary run | fitness improvement (Δ) | Hurt |
| allow new designs only as background references (ablation) | Base search · 300-design evolutionary run | fitness improvement (Δ): to 2.81% | Helped |
| remove unit-based generation (ablation) | Genesys code-generation agent · code generation proposals | valid code (%): 92% -> 73%; 20.7% drop | Hurt |
| remove unit-based generation (ablation) | Genesys code-generation agent · code generation proposals | LFC: 181 -> 75; 58.6% reduction | Hurt |
| remove symbolic checker (ablation) | Genesys code-generation agent · code generation proposals | valid code (%): 92% -> 30%; 67.4% reduction | Hurt |
| remove planner agent (ablation) | Genesys code-generation agent · code generation proposals | valid code (%): 92% -> 91% | No clear effect |
| remove observer agent (ablation) | Genesys code-generation agent · code generation proposals | valid code (%): 92% -> 89% | No clear effect |
| replace unit-based generation with direct prompting | Genesys code-generation agent · code generation proposals | valid code (%): 92% -> 6% | Hurt |
Surprising findings
- Removing the symbolic checker drops valid rate from 92% to 30%, the largest single ablation effect.
- Simple planner and observer ablations change valid rate only minimally, though the authors argue they matter qualitatively.
Limitations
- Limited to 14M-350M parameter scales; billion-parameter discovery not attempted.
- Does not integrate hardware-efficiency innovations such as FlashAttention.
- Benchmark gains are mixed; no single discovered design dominates all tasks.
Experimental setup
Datasets: SmolLM corpus 297-paper LM reference library
Benchmarks: Blimp WNLI RTE WinoGrande CoLA SST2 WSC ARC/IS MRPC
Metrics: accuracy fitness improvement Sharpe ratio maximum drawdown valid code rate
Baselines: GPT2 Mamba2 RWKV7 RetNet TTT Direct prompting
Builds on: genetic programming neural architecture search scaling laws Mamba2 RetNet
Closest papers by meaning
- Offline Training of Language Model Agents with Functions as Learnable Weights (ICML 2024 · similarity 0.89)
- Fleet of Agents: Coordinated Problem Solving with Large Language Models (ICML 2025 · similarity 0.89)
- ToolACE: Winning the Points of LLM Function Calling (ICLR 2025 · similarity 0.88)
- AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks (NeurIPS 2025 · similarity 0.88)
- LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling (NeurIPS 2026 · similarity 0.88)
- FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.