Research map / Large language models / AI and society
AI-Driven Virtual Teacher for Enhanced Educational Efficiency: Leveraging Large Pretrain Models for Autonomous Error Analysis and Correction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents VATE, a deployed virtual AI teacher that uses student draft images with multimodal and text LLMs (GPT-4o) to analyze math errors, caches results in an error pool, and runs guided dialogue on the Squirrel AI platform.
Problem
Students make errors in math problem solving, and manual error analysis by teachers is time-consuming, labor-intensive, and hard to generalize; prior LLM-based analysis from final answers alone was unreliable.
Approach
Uses student draft images as the primary evidence for error analysis via a dual-stream pipeline (multimodal model summarizes the draft, then an LLM infers error causes with problem, solution, and answer), plus an error pool keyed by question ID and wrong answer to cache analyses and bound LLM calls to O(NK).
Key results
- 78.3% of 420 selected questions passed expert review for error analysis accuracy.
- Error rate on related knowledge points dropped 37% for effective dialogue users versus non-users (per the paper's comparison).
- Ablation win rates: removing the draft gave 0.61, removing the problem 0.67, the solution 0.77, and the student answer 0.70.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VATE conversation when correcting errors | Students using VATE · Related knowledge points | error rate: 15% | Helped |
| Use highly effective VATE conversation | Students using VATE · Related knowledge points | error rate: 37% | Helped |
| Use highly effective VATE conversation | Students using VATE · Related knowledge points | NVRS: 61.5% | Helped |
| Use moderate-length dialogue instead of no dialogue | VATE learning sessions · Foundational learning sessions | average repeated learning: 9.81 -> 5.85 | Helped |
| Add student drafts to the prompt (ablation) | VATE error analysis · Error analysis | win rate: 0.61 | Helped |
| Add problem content to the prompt (ablation) | VATE error analysis · Error analysis | win rate: 0.67 | Helped |
| Add the correct solution to the prompt (ablation) | VATE error analysis · Error analysis | win rate: 0.77 | Helped |
| Add the student's answer to the prompt (ablation) | VATE error analysis · Error analysis | win rate: 0.70 | Helped |
Surprising findings
- Direct LLM analysis from final answers without drafts was reported as consistently unreliable.
Limitations
- Expert accuracy of 78.3% leaves about 21.7% of analyses below standard.
- Errors stem from both draft misinterpretation by the multimodal model and weak LLM error summarization.
- Evaluation relies on expert judgment and deployment statistics without public benchmark or released data.
Experimental setup
Datasets: Squirrel AI student drafts and answers (over 24 million drafts collected)
Metrics: expert-judged accuracy of error analysis (binary pass rate) win rate in ablation NIACT NQCT ARCT NVRS average repeated learning count
Baselines: OCR followed by LLM analysis direct LLM analysis from final answers
Builds on: GPT-4
Closest papers by meaning
- VCR: A “Cone of Experience” Driven Synthetic Data Generation Framework for Mathematical Reasoning (AAAI 2025 · similarity 0.85)
- MathMistake Checker: A Comprehensive Demonstration for Step-by-Step Math Problem Mistake Finding by Prompt-Guided LLMs (AAAI 2025 · similarity 0.84)
- Mimicking the Maestro: Exploring the Efficacy of a Virtual AI Teacher in Fine Motor Skill Acquisition (AAAI 2024 · similarity 0.83)
- EduDiag: A Benchmark for Educational Diagnostic Reasoning with Error Tracing and Correction on Large Multimodal Models (CVPR 2026 · similarity 0.83)
- Agent4Edu: Generating Learner Response Data by Generative Agents for Intelligent Education Systems (AAAI 2025 · similarity 0.83)
- Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.