Research map / Large language models / Agents, code and math
Programming by Backprop: Learning Behaviour from Symbolic Descriptions
Jonathan Cook, Silvia Sapora, Arash Ahmadian, Akbir Khan, Tim Rocktäschel, Jakob Nicolaus Foerster, Laura Ruis
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the two-stage proactive curriculum (stage 1 instructions plus executions, then stage 2 evaluation instructions) instead of either stage alone (ablation) | Proactive PBB on Qwen3-14B · Random Arithmetic | execution accuracy: negligible-to-low -> 37% | Improved |
| use RL (GRPO) instead of SFT for stage 2 of retroactive PBB (ablation) | Retroactive PBB · Random Arithmetic | generalisation to unpaired programs | Improved |
| express instructions as Python code rather than natural language (ablation) | Proactive PBB · Random Arithmetic | execution accuracy | Improved |
| add prompt context augmentation (20 templates) for instructions (ablation) | Proactive PBB · Random Arithmetic | execution accuracy | Improved |
| add chain-of-thought execution instead of direct inference | Proactive PBB · Random Arithmetic | execution accuracy | Improved |
| replace up to 100 execution examples with a single instruction in stage 2 | Proactive PBB vs algorithm distillation · Random Arithmetic | execution fidelity: 1 instruction vs up to 100 examples | No clear effect |
| add the proactive curriculum on already familiar programs (ablation) | Qwen3 models · Leetcode | execution accuracy | Improved |
| add retroactive PBB declarative stage on familiar programs (ablation) | Retroactive PBB · Leetcode | execution accuracy | No clear effect |
| mix PBB cipher instructions with skewed demonstrations | GPT-4o · Ciphers | robustness to shift values | Improved |
| add full proactive pipeline versus single stage (ablation) | Proactive PBB · Context-free grammars | grammar compliance | Improved |
| use DPO or GRPO instead of SFT for stage 2 of retroactive PBB (ablation) | Retroactive PBB on Llama-3.1-8B · Random Arithmetic | generalisation | Improved |
| use single-stage SFT mixture instead of two-stage proactive PBB with more repeated code (ablation) | Llama-3.1-8B-Instruct · Random Arithmetic | unpaired execution accuracy | No clear effect |
Closest papers by meaning
- SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning (NeurIPS 2024 · similarity 0.88)
- NExT: Teaching Large Language Models to Reason about Code Execution (ICML 2024 · similarity 0.87)
- Emergent Representations of Program Semantics in Language Models Trained on Programs (ICML 2024 · similarity 0.87)
- Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.87)
- Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation (ICML 2025 · similarity 0.86)
- Learning to Reason via Program Generation, Emulation, and Search (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
