Research map / Large language models / Agents, code and math
Can Humans Teach Machines to Code?
Celine Hocquette, Johannes Langer, Andrew Cropper, Ute Schmid
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CS-educated instead of non-CS participants as teachers | human-provided examples · six list concepts | predictive accuracy | No clear effect |
| remove erroneous examples from participants' sets (ablation) | NCS/CS human examples · dropk | predictive accuracy (%): NCS 92 -> 99; CS 66 -> 69 | Improved |
| remove erroneous examples from participants' sets (ablation) | CS human examples · append | predictive accuracy (%): 95 -> 100 | Improved |
| remove erroneous examples from participants' sets (ablation) | NCS human examples · sorted | predictive accuracy (%): 85 -> 84 | No clear effect |
| give examples from an expert in program synthesis instead of non-experts | synthesis systems · all six concepts | predictive accuracy: expert 100% for every concept vs lower non-expert averages | Improved |
| sample examples randomly instead of using NCS human examples | synthesis systems · 2 examples, NCS group | predictive accuracy: human significantly lower than uniform distribution (p=.01) | Improved |
| sample examples randomly instead of using CS human examples | synthesis systems · 2 examples, CS group | predictive accuracy: human significantly lower than both random distributions (p<.001) | Improved |
| use RandomNormal instead of CS human examples | synthesis systems · 10 examples, CS group | predictive accuracy: human lower than normal distribution | Improved |
| use CS-trained instead of NCS participants | human examples after error removal · dropk | predictive accuracy (%): NCS 99 vs CS 69 (p<.05) | Hurt |
| use more examples (non-experts gave more than expert) | human teaching · maxlist | number of examples vs accuracy: NCS 6.7 and CS 6.6 examples vs expert 3 | No clear effect |
| use longer lists in examples | expert examples · last | learned concept: single length-6 example yields last instead of fourth element | Improved |
Closest papers by meaning
- Generating Pragmatic Examples to Train Neural Program Synthesizers (ICLR 2024 · similarity 0.84)
- Are Human-generated Demonstrations Necessary for In-context Learning? (ICLR 2024 · similarity 0.83)
- Is Programming by Example Solved by LLMs? (NeurIPS 2024 · similarity 0.82)
- Students’ Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming (AAAI 2024 · similarity 0.82)
- Learning to Reason via Program Generation, Emulation, and Search (NeurIPS 2024 · similarity 0.82)
- Program Synthesis via Test-Time Transduction (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
