Research map / Large language models / Language, safety and interpretability
Automatic Short Answer Grading for Finnish with ChatGPT
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates GPT-3.5 and GPT-4 via ChatGPT API as zero-shot and one-shot graders for 2,000 Finnish university short answers across ten courses, using QWK and two new metrics (TAA, RMC).
Problem
Whether LLM chatbots can directly grade short-answer responses in Finnish university courses reliably enough for summative assessment.
Approach
Direct prompting of GPT-3.5 and GPT-4 with natural-language prompts and per-grade examples on a non-English (Finnish) ASAG dataset, plus two new end-user-oriented metrics (tolerance-adjusted accuracy and relative merit consensus) alongside QWK.
Key results
- One-shot GPT-4 reaches QWK of at least 0.6 on 44 of 100 questions, versus 21 for one-shot GPT-3.5 and 8 for zero-shot GPT-3.5.
- One-shot GPT-4 reaches zero-tolerance accuracy of at least 0.8 on only 17% of questions, but performs well on binary pass/fail questions (21 of 25 at least 0.75 accuracy).
- A weak negative association exists between answer length and QWK; QWK correlates with prediction standard deviation (Pearson 0.39, p=6.6e-05) for one-shot GPT-4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use one-shot examples instead of zero-shot prompting | GPT-3.5 · 100 Finnish short-answer questions | questions with QWK ≥0.6: 8 -> 21 | Helped |
| Use one-shot examples instead of zero-shot prompting | GPT-4 · 100 Finnish short-answer questions | questions with QWK ≥0.6: 19 -> 44 | Helped |
| Use GPT-4 instead of GPT-3.5 | Zero-shot grading · 100 Finnish short-answer questions | questions with QWK ≥0.6: 8 -> 19 | Helped |
| Use GPT-4 instead of GPT-3.5 | One-shot grading · 100 Finnish short-answer questions | questions with QWK ≥0.6: 21 -> 44 | Helped |
| Increase average student-answer length | GPT-4 one-shot grading · Finnish short-answer questions | QWK | Hurt |
| Use one-shot examples instead of zero-shot prompting | ChatGPT grading · Finnish short-answer questions | model performance | Helped |
Surprising findings
- Zero-shot GPT-3.5 often assigned a single uniform score to all answers for long-answer questions, yielding QWK of 0.
- Models were more lenient than human graders on binary and most numerical scales, yet still assigned fewer fails than teachers in the one-shot setting.
Limitations
- Non-English (Finnish) data, where ChatGPT tokenization is inefficient (about 2.11 cl100k tokens per FinBERT token) and performance may be lower.
- No grading criteria or reference answers available, constraining prompt content; answers may have been graded by different evaluators.
- Metrics are sensitive to the number of possible scores, and no few-shot smaller-model baseline (e.g., BERT) was compared.
Experimental setup
Datasets: University of Turku Finnish short-answer exam data (2,000 test answers, 377 example answers, ten courses)
Benchmarks: University of Turku Finnish short-answer grading dataset
Metrics: quadratic weighted kappa tolerance-adjusted accuracy relative merit consensus
Baselines: majority baseline
Builds on: Haller et al. 2022 ASAG survey chain-of-thought prompting
Closest papers by meaning
- LLMs in the Classroom: Outcomes and Perceptions of Questions Written with the Aid of AI (AAAI 2025 · similarity 0.86)
- Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models (ICLR 2024 · similarity 0.83)
- ChatQA: Surpassing GPT-4 on Conversational QA and RAG (NeurIPS 2024 · similarity 0.82)
- Students’ Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming (AAAI 2024 · similarity 0.81)
- Understanding GenAI for Teaching and Learning in Secondary Classrooms (AAAI 2025 · similarity 0.81)
- Language Models as Science Tutors (ICML 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.