Research map / Large language models / Agents, code and math
Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates six deployed coding agents on four esoteric languages (EsoLang-Bench) in a sequential agentic protocol, finding strong agents succeed by metaprogramming (Python generators emitting Brainfuck/Befunge code), and that helper-library scaffolds transfer this to mid-tier agents.
Problem
Mainstream coding benchmarks (SWE-Bench Verified, Terminal-Bench 2.0) compress coding agents into narrow bands and do not test how agents adapt when the programming language itself is unfamiliar.
Approach
Use esoteric languages as controlled proxies for unfamiliar executable interfaces, run deployed coding agents in a sequential session with local execution and hidden-test submissions, and causally test metaprogramming (host-language generators) via a no-metaprogramming ablation and transfer of executable helper libraries versus written advice.
Key results
- Opus 4.6 and GPT-5.4 xhigh solve 86.9% and 99.7% mean on EsoLang-Bench, while Kimi K2.5 and Haiku 4.5 reach 11.3% and 24.7%.
- Forbidding metaprogramming drops Opus 4.6 and GPT-5.4 xhigh sharply on Brainfuck and Befunge-98 (e.g., Opus 64/80 to 27/80 direct authoring on Brainfuck).
- A reference library of generator helpers lifts Sonnet 4.6 on Brainfuck from 12/80 to 64/80 and GPT-5.4 mini from 5/80 to 53/80, while written text guidance yields little gain.
Surprising findings
- Written strategy guidance distilled from the strongest agent's traces does not materially improve weaker agents, while executable helper code does.
- More interpreter calls and output tokens help only agents that can already use feedback; weaker agents stay near floor.
Limitations
- Closed-source strongest agents prevent inspection of training exposure to esolangs.
- Esoteric languages are artificial proxies, not production targets.
- Results depend on harness (Claude Code, Codex, OpenCode) which are not internally identical; metaprogramming claim limited to Brainfuck and Befunge-98.
Experimental setup
Datasets: EsoLang-Bench
Benchmarks: EsoLang-Bench
Metrics: solved problems out of 80 Wilson 95% confidence interval
Baselines: SWE-Bench Verified Terminal-Bench 2.0 LiveCodeBench v6
Builds on: EsoLang-Bench SWE-bench LiveCodeBench
Closest papers by meaning
- EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions (NeurIPS 2026 · similarity 0.88)
- CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments (ICML 2026 · similarity 0.88)
- NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.88)
- Hybrid-Gym: Training Coding Agents to Generalize Across Tasks (ICML 2026 · similarity 0.88)
- Toward Interactive Understanding of Code APIs (NeurIPS 2026 · similarity 0.88)
- Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.