Research map / Large language models / Agents, code and math
Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a benchmark of elementary visual programming computational thinking tests (HoC, ACE, CT-Test) for generative models and fine-tunes Llama3-8B on a symbolically generated synthetic dataset to approach GPT-4o performance.
Problem
Generative models struggle with simple computational thinking and problem-solving tasks that elementary school students can solve, and there is no benchmark comparing models to students on visual programming tests.
Approach
Build a symbolic, execution-verified synthetic data generator for solution synthesis and multi-choice programming questions at multiple skill levels (basics, tracing, grid synthesis), with trace-based explanations produced by an emulator rather than by a stronger LLM, and fine-tune Llama3-8B with LoRA.
Key results
- GPT-4o with combined text and visual input is reported to barely match the average school student, with Llama3-8B performing worse.
- The best fine-tuned Llama3-8B variant (HierFinetune) achieves overall results comparable to GPT-4o combined input, with at least 10% gains from fine-grained skill data.
- Models match grade 7 students on ACE but lag behind grade 3 students on HoC solution synthesis tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine text and visual grid representations instead of using either modality alone (ablation) | GPT-4o · HoC, ACE, and CT-Test | accuracy | Helped |
| train with symbolic execution-derived explanations instead of answer-only targets (ablation) | Llama3-8B fine-tuned on solution synthesis tasks · HoC | accuracy | Helped |
| train with explanations on solution synthesis and MCQ tasks instead of answer-only targets (ablation) | Llama3-8B fine-tuned on solution synthesis and MCQ tasks · ACE and CT-Test | accuracy | Helped |
| add fine-grained skills data to fine-tuning (ablation) | Llama3-8B fine-tuned on the synthetic dataset · HoC, ACE, and CT-Test | accuracy: at least 10% | Helped |
| train with explanations across the full synthetic dataset, including fine-grained skills (ablation) | Llama3-8B fine-tuned on the full synthetic dataset · HoC, ACE, and CT-Test | accuracy | Helped |
| provide correct reasoning in the inference prompt (ablation) | Llama3-8B fine-tuned on the full synthetic dataset · HoC, ACE, and CT-Test | accuracy | Helped |
| use few-shot prompting instead of zero-shot chain-of-thought prompting (ablation) | GPT-family models · HoC, ACE, and CT-Test | accuracy | No clear effect |
| fine-tune on the synthetic dataset instead of using the unfine-tuned model (ablation) | Llama3-8B · HoC-synth, HoC-synth hard, HoC-Filtered, HoC-OoD, and ACE-synth | accuracy | Helped |
| fine-tune on the synthetic dataset rather than use GPT-4oCombined | Llama3-8B · HoC-Filtered | accuracy | Helped |
| fine-tune on the synthetic dataset rather than use GPT-4oCombined | Llama3-8B · HoC-OoD | accuracy | No clear effect |
Surprising findings
- Few-shot prompting did not improve results over zero-shot CoT for the evaluated models.
- Llama3-8B often guesses answers without reasoning, and GPT-4o struggles with grid layouts such as missing walls.
Limitations
- Multimodal models were evaluated but not fine-tuned.
- Best fine-tuned variant uses correct explanations at inference, which are not available in practice (oracle setting).
Experimental setup
Datasets: HoC (Hour of Code: Maze Challenge) ACE test CT-Test Synthetic training data generated from HoC/Karel-style codes and grids
Benchmarks: HoC ACE CT-Test HoC-synth ACE-synth HoC-OoD
Metrics: accuracy reasoning correctness (manual annotation)
Baselines: GPT-4o GPT-4 GPT-3.5 CodeLlama-7B LLaVA-7B Llama3-8B Chance baseline School students (average grade 3-7)
Builds on: LoRA Chain-of-thought prompting Bloom's taxonomy
Closest papers by meaning
- Measuring Vision-Language STEM Skills of Neural Models (ICLR 2024 · similarity 0.87)
- GENOME: Generative Neuro-Symbolic Visual Reasoning by Growing and Reusing Modules (ICLR 2024 · similarity 0.87)
- DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback (ICLR 2025 · similarity 0.87)
- IDEA-Bench: How Far are Generative Models from Professional Designing? (CVPR 2025 · similarity 0.87)
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (ICML 2024 · similarity 0.87)
- Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.