Research map / Large language models / AI and society
Does GPT Really Get It? A Hierarchical Scale to Quantify Human and AI’s Understanding of Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a five-level hierarchy for understanding algorithms and uses it to compare undergraduate/graduate students with GPT-3.5, GPT-4-turbo and GPT-4o on the Euclidean and Ford-Fulkerson algorithms.
Problem
Existing work lacks a rigorous, functional definition and scale for measuring whether LLMs understand algorithms, as opposed to merely producing correct outputs, and how this compares to human understanding.
Approach
A functional, algorithmically testable hierarchy of understanding (execution, step-by-step evaluation, representation, exemplification/explanation, extrapolation/counterfactual reasoning) grounded in Understanding as Representation Manipulability, applied identically to humans and GPT via a survey with randomized question versions.
Key results
- GPT-4 and GPT-4o improved over GPT-3.5 on every question (p<0.05), and GPT-4 scored close to doctoral students, though doctoral students beat GPT-4 on Level 5 extension questions.
- All GPT versions performed better on natural-language tasks than mathematical tasks for Ford-Fulkerson (p<0.05), while students showed no such gap.
- Adding a correct example response improved GPT-3.5 accuracy on max flow but had little effect on GPT-4o and slightly decreased GPT-4 accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 instead of GPT-3.5 | GPT on algorithm-understanding questions · Euclidean algorithm and Ford-Fulkerson | question score | Helped |
| Use GPT-4o instead of GPT-3.5 | GPT on algorithm-understanding questions · Euclidean algorithm and Ford-Fulkerson | question score | Helped |
| Use GPT-4o instead of GPT-4 | GPT on algorithm-understanding questions · Euclidean algorithm and Ford-Fulkerson | overall score | No clear effect |
| Advance from undergraduate to doctoral study | Students answering algorithm-understanding questions · Euclidean algorithm and Ford-Fulkerson | question score | Helped |
| Use a correct example response in chat history (ablation) | GPT-3.5 on Max Flow problems · Max Flow trivial and intermediate problems | accuracy | Helped |
| Use a correct example response in chat history (ablation) | GPT-4o on Max Flow problems · Max Flow trivial and intermediate problems | accuracy | No clear effect |
| Use a correct example response in chat history (ablation) | GPT-4 on Max Flow problems · Max Flow trivial and intermediate problems | accuracy | Hurt |
| Reason about language rather than mathematical tasks | GPT versions on algorithm-understanding questions · Ford-Fulkerson, Levels 4 and 5 | score | Helped |
| Reason about language rather than mathematical tasks | GPT versions on algorithm-understanding questions · Euclidean algorithm, Level 4 | score | Helped |
| Reason about language rather than mathematical tasks | GPT versions on algorithm-understanding questions · Euclidean algorithm, Level 5 | score | No clear effect |
| Answer coding tasks rather than evaluation tasks | GPT · Euclidean algorithm and Ford-Fulkerson | score | Helped |
| Answer coding tasks rather than evaluation tasks | Students · Euclidean algorithm and Ford-Fulkerson | score | Hurt |
Surprising findings
- GPT models nearly perfect on code generation tasks, outperforming their evaluation-question performance, the reverse of the student trend.
- GPT repeatedly produced hedged, incorrect statements and hallucinated counterexamples to true statements, behaviors not seen in humans.
Limitations
- Small human sample (n=23 after exclusions), so comparisons with GPT are confounded by fatigue and population size.
- Only two algorithms tested, and GPT results depend on a limited set of API versions.
- Hand-graded scoring of GPT responses and explanation rubric involve subjectivity.
Experimental setup
Datasets: randomized Euclidean GCD and Ford-Fulkerson survey questions
Benchmarks: Euclidean algorithm survey Ford-Fulkerson max flow survey
Metrics: mean score out of two per question accuracy 95% confidence intervals
Baselines: GPT-3.5-turbo GPT-4-turbo undergraduate students graduate students
Builds on: Understanding as Representation Manipulability (URM) Bloom's Taxonomy chain-of-thought prompting
Closest papers by meaning
- Position: We Need An Algorithmic Understanding of Generative AI (ICML 2025 · similarity 0.89)
- CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.87)
- When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaboration (NeurIPS 2025 · similarity 0.87)
- A Matter of Interest: Understanding Interestingness Judgments of Math Problems in Humans and Language Models (NeurIPS 2026 · similarity 0.87)
- Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choice (ICLR 2025 · similarity 0.87)
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.