Research map / Large language models / Agents, code and math
CodeIt: Self-Improving Language Models with Prioritized Hindsight Replay
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CodeIt, an expert-iteration loop that samples DSL programs from a CodeT5+ policy, relabels goals with realized outputs (hindsight), and trains with prioritized replay, solving 59/400 ARC evaluation tasks.
Problem
Solving ARC as programming-by-examples faces extremely sparse rewards and poor inter-task generalization; prior LLM and symbolic methods either don't use experience across tasks or scale poorly.
Approach
Combine expert iteration with hindsight relabeling (replace target outputs with realized program outputs so every valid program yields training data) and prioritized replay weighted by demonstration correctness, to avoid catastrophic forgetting while learning from all samples.
Key results
- CodeIt solves 59/400 ARC evaluation tasks (pass@3), outperforming the mutation d1 baseline (42/400) and prior symbolic and LLM baselines (e.g., 27/400 text-davinci-003).
- On ARC Eval 412 (pass@1), CodeIt solves 59/412 versus 49/412 for GPT-4-based approach.
- Ablation: removing hindsight relabeling (A2) yields 42/400 cumulative and removing priority (A3) drops policy-only performance from 49/400 to 38/400.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CodeIt instead of the Mutation d1 baseline | ARC program synthesis · ARC Eval Set | pass@3 solved tasks: 42 / 400 -> 59 / 400 | Helped |
| Add policy feedback through Expert Iteration (ablation) | CodeIt · ARC evaluation | cumulative solved tasks: 45/400 -> 59/400 | Helped |
| Add hindsight relabeling instead of retaining only fully correct demonstration solutions (ablation) | CodeIt · ARC evaluation | cumulative solved tasks: 42/400 -> 59/400 | Helped |
| Prioritize replay-buffer experiences instead of sampling uniformly (ablation) | CodeIt · ARC evaluation | policy-only solved tasks: 38/400 -> 49/400 | Helped |
| Use a pretrained policy instead of randomly initialized weights (ablation) | CodeIt · ARC evaluation | cumulative solved tasks: 35/400 -> 59/400 | Helped |
| Use multiple demonstration examples rather than at most one (ablation) | CodeIt · ARC evaluation | cumulative solved tasks: 51/400 -> 59/400 | Helped |
| Add mutation-based training data augmentation (ablation) | CodeIt · ARC evaluation | cumulative solved tasks: 20/400 -> 59/400 | Helped |
Surprising findings
- Learning only from mutation experiences (A1, no policy feedback) still improves over the mutation baseline but is far below CodeIt, showing policy feedback matters.
- Full-parameter finetuning of CodeT5 outperforms LoRA-based finetuning of Mistral-7B, which reached under 9% solved tasks, versus about 11% for CodeT5 220M.
Limitations
- Relies on hand-designed components: a DSL, an interpreter, and ground-truth training programs.
- Cannot solve tasks whose ground-truth programs exceed about 11 lines; some primitives are never learned.
- Context truncation limits representation of large grids and many demonstration examples.
Experimental setup
Datasets: ARC training split ARC evaluation split ConceptARC
Benchmarks: ARC evaluation set (400 tasks) ARC Eval 412 ARC custom validation split ConceptARC
Metrics: pass@3 pass@1 cumulative performance policy performance
Baselines: GPT-4 direct prediction (Ho et al. / Bubeck-style LLM ARC) text-davinci-003 ARC LLM Mutation d1 baseline Mutation dinf baseline Random program baseline Icecuber-style symbolic search DreamCoder-style search
Builds on: expert iteration hindsight experience replay DreamCoder Hodel ARC DSL
Closest papers by meaning
- Self-Improvement in Language Models: The Sharpening Mechanism (ICLR 2025 · similarity 0.87)
- Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective (ICML 2025 · similarity 0.86)
- Improving Reasoning Performance in Large Language Models via Representation Engineering (ICLR 2025 · similarity 0.86)
- CodePlan: Unlocking Reasoning Potential in Large Language Models by Scaling Code-form Planning (ICLR 2025 · similarity 0.86)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.86)
- The Surprising Effectiveness of Test-Time Training for Few-Shot Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.