Research map / Large language models / AI and society
Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks seven LVLMs (GPT-4o, Gemini, Claude-3, open-source models) on SMART-840, 840 Mathematical Kangaroo puzzles from 2020-2024 across grades 1-12, comparing their accuracy and response correlations with children's performance.
Problem
Whether large vision-and-language models solve visuo-linguistic math and algorithmic puzzles with the grade-progressive reasoning that children display, and how their performance compares to children at each grade level.
Approach
Uses all Mathematical Kangaroo olympiad questions from 2020-2024 (grades 1-12) with official children's performance statistics as a human reference, enabling grade-wise, difficulty-index, discriminative-index, and rank-based comparisons of LVLMs against children in zero-shot evaluation.
Key results
- GPT-4o achieves 42.5% average accuracy on SMART-840 versus 64.2% for children; Claude-3 Sonnet reaches 49.7%, the highest among the evaluated models.
- Gemini-Pro accuracy rises from about 25% on grades 1-2 to about 40% on grades 11-12, showing better performance on higher-grade problems.
- Correlations between LVLM and children's difficulty indices are weak, and discriminative-index correlations are mostly negative.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Claude-3 Sonnet instead of GPT-4o | LVLM evaluation on SMART-840 · SMART-840 | accuracy (%): 42.4 -> 49.7 | Helped |
| Use GPT-4o instead of GPT-4v | LVLM evaluation on SMART-840 · SMART-840 | accuracy (%): 37.4 -> 42.4 | Helped |
| Use Gemini-Pro instead of Gemini-Flash | LVLM evaluation on SMART-840 · SMART-840 | accuracy (%): 29.4 -> 31.7 | Helped |
| Use GPT-4o instead of Random | SMART-840 answering · SMART-840 | accuracy (%): 20.1 -> 42.4 | Helped |
| Use GPT-4o instead of Human | SMART-840 answering · SMART-840 | accuracy (%): 64.2 -> 42.4 | Hurt |
| Evaluate on text-only rather than image-text problems | GPT-4o · SMART-840 | accuracy (%): 36.45 -> 92.31 | Helped |
| Evaluate on text-only rather than image-text problems | Claude-Sonnet · SMART-840 | accuracy (%): 45.79 -> 100.00 | Helped |
| Evaluate on text-only rather than image-text problems | Gemini-Pro · SMART-840 | accuracy (%): 20.56 -> 69.23 | Helped |
| Remove explanation request from the prompt (ablation) | GPT-4o · Number category | accuracy (%): 49.5% to 17.6% | Hurt |
| Remove explanation request from the prompt (ablation) | GPT-4o · Algebra category | accuracy (%): 63.4% to 31.5% | Hurt |
| Remove explanation request from the prompt (ablation) | GPT-4o · SMART-840 text-only problems | accuracy (%) | Hurt |
| Use OpenAI o1-preview instead of GPT-4o | Text-only problem evaluation · SMART-840 text-only problems, grades 1 & 2 | accuracy (%): 92.31 -> 100.00 | Helped |
Surprising findings
- LVLMs perform better on higher-grade problems than on lower-grade problems, the reverse of the children's progression.
- GPT-4o accuracy drops sharply when asked to answer without explanation (e.g., from 49.5% to 17.6% on the number category).
Limitations
- The dataset is small (840 problems), and the authors state that the size is necessary given the goal of zero-shot evaluation.
- Repeated runs were limited by API cost, so only about 2-5 runs per problem were made, with substantial run-to-run variance.
- Manual validation was applied only to GPT-4o responses, and unparseable responses are counted as incorrect.
Experimental setup
Datasets: Mathematical Kangaroo 2020-2024 exams
Benchmarks: SMART-840
Metrics: accuracy standard deviation Pearson correlation with difficulty index discriminative index correlation national rank and percentile
Baselines: Random Gemini-Flash GPT-4v Claude-3 Opus Claude-3 Sonnet XGen-MM-Phi3-v1 InternVL-Chat-V1.2 InternLM-XComposer2 LLaVA-NeXT
Builds on: SMART-101 MathVista
Closest papers by meaning
- Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems (AAAI 2025 · similarity 0.90)
- OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI (NeurIPS 2024 · similarity 0.89)
- We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning (ICLR 2026 · similarity 0.89)
- Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process (ICLR 2025 · similarity 0.89)
- DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models (ICLR 2025 · similarity 0.89)
- CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.