Research map / Large language models / Agents, code and math
From I/O to Code with Discovery Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIO-Agent, an LLM-driven evolutionary search that uses the Transformation Priority Premise as a mutation prior and curriculum-wise evolution to synthesize programs from input-output examples (IO2Code), and introduces IO2CodeBench.
Problem
Synthesizing programs purely from input-output behavior (IO2Code), without natural-language specifications, requires inductive discovery and avoids overfitting and aimless search in an underspecified program space.
Approach
Frames IO2Code as LLM-driven evolutionary search guided by the Transformation Priority Premise as a mutation prior (simplest constructs first), a curriculum over visible examples sorted easy to hard, and error-grounded feedback from concrete failing cases.
Key results
- DIO-Agent reaches 58.63% average pass rate on IO2CodeBench, versus 49.60% for the strongest baseline CodeEvolve.
- On the Algorithm level DIO-Agent achieves 71.43%, and on Multimodal 40.67% versus 22.00% for AlphaEvolve.
- Against test-time scaling with N=40, Best-of-N reaches 40.28% average while DIO-Agent reaches 63.66%.
Surprising findings
- Self-Consistency performs worse than Best-of-N, suggesting the correct rule is often not the most frequent one under the model's prior.
- Many failed runs achieved perfect visible-set accuracy but failed held-out tests, and were much longer programs.
Limitations
- Relies on API-accessed closed LLMs; evaluation limited to three backbones.
- Generalization checks depend on held-out tests; visible-set overfitting remains a failure mode.
- Small benchmark sizes per level (e.g., about 30 tasks in the stratified analysis) lead to noisy estimates.
Experimental setup
Datasets: IO2CodeBench
Benchmarks: IO2CodeBench
Metrics: pass rate (all held-out tests) token/iter average pass rate across levels
Baselines: Direct prompting PBE E-PBE TDD-guided Generation FunSearch AlphaEvolve CodeEvolve Best-of-N Self-Consistency
Builds on: AlphaEvolve FunSearch Transformation Priority Premise programming by example
Closest papers by meaning
- Simple Baselines are Competitive with Code Evolution (NeurIPS 2026 · similarity 0.90)
- CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments (ICML 2026 · similarity 0.88)
- Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGI (ICML 2025 · similarity 0.88)
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.87)
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.87)
- R2E: Turning any Github Repository into a Programming Agent Environment (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.