Research map / Vision-language and multimodal / Multimodal LLMs
Measuring Vision-Language STEM Skills of Neural Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces STEM, a ~1.07M-question multimodal K-12 benchmark spanning science, technology, engineering, and math, and evaluates CLIP, GPT-3.5-Turbo, and other models against elementary students.
Problem
Existing vision-language benchmarks cover only one STEM subject or convert images to text, so the multimodal STEM skills of neural models at fundamental (K-12) level are not well measured.
Approach
A large multimodal multiple-choice STEM benchmark organized by fine-grained K-12 skills and grades, scored with IXL's SmartScore exam system and compared against human students, plus CLIP finetuning baselines.
Key results
- Zero-shot CLIP RN50x64 reaches about 54.9% average accuracy, versus 49.0% for GPT-3.5-Turbo and 37.6% for GloVe, with random guessing at 36.9%.
- Finetuning CLIP ViT-L/14@336px raises average accuracy to 76.3%, with science 87.0% and math 78.4%.
- Average elementary-grade exam score is 40.8, about 54.7% below the human reference of 90.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Finetune CLIP on the full subject training set | CLIP ViT-L/14@336px · STEM test, Average | accuracy: 54.4 -> 76.3 | Helped |
| Train with 16 samples per skill instead of zero-shot (ablation) | CLIP ViT-L/14@336px · STEM test, Average | accuracy: 54.4 -> 67.8 | Helped |
| Train with 1 or 2 samples per skill (ablation) | CLIP in Technology · STEM test, Technology | accuracy | Hurt |
| Train one model on the mixture of all four subject training sets instead of individual subject models (ablation) | CLIP ViT-L/14@336px · STEM test, Engineering | accuracy: 67.7 -> 73.4 | Helped |
| Train one model on the mixture of all four subject training sets instead of individual subject models (ablation) | CLIP ViT-L/14@336px · STEM test, Technology | accuracy: 71.9 -> 60.4 | Hurt |
| Train one model on the mixture of all four subject training sets instead of individual subject models (ablation) | CLIP ViT-L/14@336px · STEM test, Science | accuracy: 87.0 -> 86.3 | Hurt |
| Train one model on the mixture of all four subject training sets instead of individual subject models (ablation) | CLIP ViT-L/14@336px · STEM test, Math | accuracy: 78.4 -> 77.7 | Hurt |
| Change zero-shot prompt format from Q+A to A+Q (ablation) | CLIP · STEM test, Average | accuracy: 54.4 -> 52.2 | Hurt |
| Change zero-shot prompt format from Q+A to Q followed by 'Choose the best answer:' and A (ablation) | CLIP · STEM test, Average | accuracy: 54.4 -> 53.7 | Hurt |
Surprising findings
- No obvious performance drop with increasing grade level for neural models, unlike human learning curves.
- Zero-shot CLIP is overconfident and poorly calibrated.
Limitations
- Current models remain far below elementary-student performance even after finetuning.
- Many models are evaluated zero-shot with an ad hoc matching setup, and human comparison uses a subset of skills due to constraints.
- Some models (GPT-3.5-Turbo, text-davinci-002) are accessed via API and may change over time.
Experimental setup
Datasets: IXL Learning (science, engineering, math) ProProfs Quizzes and Triviaplaza (technology) ScienceQA IconQA Geometry3K
Benchmarks: STEM
Metrics: accuracy macro average accuracy IXL SmartScore exam score
Baselines: GloVe UnifiedQA GPT-3 GPT-3.5-Turbo Virtex 12-in-1 ViLBERT UNITER
Builds on: CLIP ScienceQA UnifiedQA
Closest papers by meaning
- Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming (NeurIPS 2024 · similarity 0.87)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.86)
- CodePercept: Code-Grounded Visual STEM Perception for MLLMs (CVPR 2026 · similarity 0.85)
- DevBench: A multimodal developmental benchmark for language learning (NeurIPS 2024 · similarity 0.85)
- Alignment of CNN and Human Judgments of Geometric and Topological Concepts (AAAI 2025 · similarity 0.85)
- RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.