Research map / Large language models / Reasoning and chain of thought
A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students’ Formative Assessment Responses in Science
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses GPT-4 with few-shot chain-of-thought prompting and human-in-the-loop active learning to score and explain middle school Earth Science formative assessment responses.
Problem
Automated scoring of open-ended, short-answer K-12 science formative assessments, which existing methods (built for structured math/CS tasks) handle poorly and which lack explanations for scores.
Approach
Combines few-shot in-context examples with chain-of-thought rationales (evidence + rubric reference + score) and a human-in-the-loop active learning loop that inserts error-correcting validation instances into the prompt, so GPT-4 both scores and explains responses aligned with human consensus.
Key results
- Few-shot CoT with active learning (CoT + AL) gave the best Q2 total-score QWK of 0.87 and Q3 total-score QWK of 0.90 among the four implementations.
- Q1 Arrow Size: Few-Shot (no CoT) reached F1 1.00 and QWK 1.00 on the test set, while CoT + AL reached F1 0.97 and QWK 0.95.
- Of 11 subscores and total scores, 9 reached QWK >= 0.8 at some point and 4 reached QWK > 0.9; CoT and active learning showed overfitting in some subscores.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adding CoT reasoning to few-shot examples (ablation) | GPT-4 scoring of Q1 Arrow Size · Q1 Arrow Size | Macro F1 and QWK: F1 1.00 to 0.95; QWK 1.00 to 0.89 | Hurt |
| Adding active learning on top of CoT (ablation) | GPT-4 scoring of Q1 Arrow Size · Q1 Arrow Size | Macro F1 and QWK: F1 0.95 to 0.97; QWK 0.89 to 0.95; still below few-shot without CoT | Mixed |
| Adding CoT reasoning to few-shot examples (ablation) | GPT-4 scoring of Q2 total score · Q2 Total Score | QWK: QWK 0.55 to 0.80 | Helped |
| Adding CoT and active learning to few-shot prompting (ablation) | GPT-4 scoring of Q2 Arrow Size subscore · Q2 Arrow Size | Macro F1: F1 0.69 to 0.92 | Helped |
| Adding CoT reasoning to few-shot examples (ablation) | GPT-4 scoring of Q2 Arrow Direction Reasoning · Q2 Arr. Dir., Reasoning | QWK: QWK 0.47 zero-shot to 0.41 with CoT, and 0.30 after active learning | Hurt |
| Adding few-shot examples (numerical scores only) (ablation) | GPT-4 scoring of Q3 subscores · Q3 Arrow Size and Runoff Direction | QWK: Q3 Arrow Size QWK 0.67 to 0.73; Runoff Direction QWK 0.77 to 0.80 | Helped |
| Adding CoT reasoning to few-shot examples (ablation) | GPT-4 scoring of Q3 Arrow Size subscores · Q3 Arrow Size | QWK: QWK 0.73 to 0.65 | Hurt |
| Adding CoT reasoning to few-shot examples (ablation) | GPT-4 scoring of Q3 Runoff Direction · Q3 Runoff Direction | QWK: QWK 0.80 to 0.84 | Helped |
| Adding active learning on top of CoT (full CoT + AL method) (ablation) | GPT-4 scoring of Q3 Arrow Size subscores · Q3 Arrow Size | QWK: QWK 0.65 to 0.83 | Helped |
| Adding active learning on top of CoT (ablation) | GPT-4 scoring of Q3 Runoff Direction · Q3 Runoff Direction | QWK: QWK 0.84 to 0.75 | Hurt |
| Full CoT + active learning pipeline versus zero-shot on Q2 total score (ablation) | GPT-4 scoring of Q2 total score · Q2 Total Score | QWK: QWK 0.65 to 0.87 | Helped |
Surprising findings
- Adding CoT reasoning often decreased performance relative to plain few-shot prompting (e.g., Q1 Arrow Size, Q3 Arrow Size), and the plain few-shot setting sometimes outperformed the full method.
- Active learning and CoT led to overfitting, especially on simpler subproblems and ambiguous reasoning subscores.
Limitations
- Small, imbalanced dataset from one curriculum and three questions; results may not generalize.
- CoT and active learning caused overfitting, and only one active learning iteration was run.
- Rubric was not updated during active learning; ethical concerns about privacy, bias and hallucination in classrooms.
Experimental setup
Datasets: SPICE middle school Earth Science formative assessment responses (3 questions, 270 students)
Benchmarks: SPICE formative assessment test set (held-out 20%)
Metrics: Macro F1 Cohen's quadratic weighted kappa accuracy
Baselines: Zero-Shot GPT-4 Few-Shot GPT-4 Few-Shot, CoT GPT-4
Builds on: chain-of-thought prompting in-context learning active learning persona pattern
Closest papers by meaning
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (ICML 2024 · similarity 0.85)
- LLMs in the Classroom: Outcomes and Perceptions of Questions Written with the Aid of AI (AAAI 2025 · similarity 0.84)
- K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs (NeurIPS 2026 · similarity 0.84)
- T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Large Language Model Signals for Science Question Answering (AAAI 2024 · similarity 0.83)
- CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.83)
- Scaling Equitable Reflection Assessment in Education via Large Language Models and Role-Based Feedback Agents (AAAI 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.