Research map / Multi-agent systems and games / LLM agents
Lessons Learned: A Multi-Agent Framework for Code LLMs to Learn and Improve
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LessonL, a multi-agent framework where small code LLMs generate, bank, and select natural-language lessons from each other's successes and failures to iteratively improve code optimization and generation.
Problem
Code optimization (and generation) where no single LLM dominates across problem categories and complementary strengths are unknown a priori; how to make multiple LLM agents learn from each other without pre-specified roles.
Approach
Agents solicit concise natural-language lessons from comparing their code to the original (speedup, slowdown, incorrectness, syntax error), deposit them in a shared bank, and select lessons by a speedup-times-adjustment-factor score plus code-relevance retrieval, iterating for several rounds.
Key results
- On ParEval OpenMP mode, LessonL reaches 3.46 geometric mean speedup, close to o3 (3.55) and above GPT-4o (2.93), using three small open-source agents.
- On ParEval serial mode, LessonL achieves 0.91 correctness and 2.16 speedup, beating GPT-4o (0.80, 1.72).
- On code generation, LessonL attains pass@1 of 0.915 on HumanEval and 0.899 on MBPP, best among compared methods on three of four benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LessonL instead of OpenAI o3 | Code optimization · ParEval serial mode | speedup: 2.21 $\pm$ 0.16 -> 2.16 $\pm$ 0.11 | Hurt |
| use LessonL instead of OpenAI o3 | Code optimization · PolyBench OpenMP mode | speedup: 3.15 $\pm$ 0.23 -> 3.40 $\pm$ 0.12 | Helped |
| use lesson selection by both speedup and relevance with dynamic adjustment rather than only high-speedup selection (ablation) | LessonL agents · ParEval OpenMP mode | speedup: 3.20 $\pm$ 0.04 -> 3.46 $\pm$ 0.03 | Helped |
| use lesson selection by both speedup and relevance with dynamic adjustment rather than only high-relevance selection (ablation) | LessonL agents · ParEval serial mode | speedup: 1.96 $\pm$ 0.08 -> 2.16 $\pm$ 0.11 | Helped |
| add dynamic speedup adjustment (ablation) | LessonL agents · ParEval OpenMP mode | speedup: 3.28 $\pm$ 0.10 -> 3.46 $\pm$ 0.03 | Helped |
| use prioritized lesson selection rather than random selection (ablation) | LessonL agents · ParEval serial mode | speedup: 2.03 $\pm$ 0.05 -> 2.16 $\pm$ 0.11 | Helped |
| add lessons (ablation) | LessonL agents · ParEval OpenMP mode | speedup: 3.01 $\pm$ 0.16 -> 3.46 $\pm$ 0.03 | Helped |
| use LessonL rather than MoA | Code generation · HumanEval | pass@1: 0.909 -> 0.915 | Helped |
| use LessonL rather than MoA | Code generation · MBPP+ | pass@1: 0.770 -> 0.765 | Hurt |
Surprising findings
- Reflexion and LessonL keep improving with more rounds, while MoA performance drops below GPT-4o beyond three rounds despite using GPT-4o as aggregator.
- Scaling inference-time sampling (CoT(20), Qwen14B(20)) does not help, and multi-agent methods like MapCoder and MoA are the least cost-effective.
Limitations
- Lessons add overhead to time to first token, deferring the response.
- Applied only to function-level snippets, not repository-level tasks like SWE-bench.
- Agents rely on fixed hyperparameters and heuristic lesson selection; autonomy in lesson selection is left open.
Experimental setup
Datasets: ParEval PolyBench HumanEval HumanEval+ MBPP MBPP+
Benchmarks: ParEval PolyBench HumanEval HumanEval+ MBPP MBPP+
Metrics: geometric mean speedup correctness rate proportion with >2x speedup pass@1
Baselines: Qwen14B GPT-4o mini GPT-4o DeepSeek-R1-14B OpenAI o3 CoT Reflexion MapCoder MoA
Builds on: Reflexion MoA MapCoder Chain-of-thought Retrieval-augmented generation
Closest papers by meaning
- RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.88)
- CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments (ICML 2026 · similarity 0.87)
- Credit-Budgeted ICPC-Style Coding: When LLM Agents Must Pay for Every Decision (ICLR 2026 · similarity 0.87)
- Generalizable End-to-End Tool-Use RL with Synthetic CodeGym (ICLR 2026 · similarity 0.87)
- TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination (ICML 2026 · similarity 0.87)
- CORVUS: Context Optimization and Reduction Via Underlying Synchronization for LLM Coding Agents (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.