Research map / Large language models / AI and society
LLMs in the Classroom: Outcomes and Perceptions of Questions Written with the Aid of AI
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Randomly deploys instructor-written and ChatGPT-4o-written (human-validated) multiple-choice questions in an operating systems course, comparing student scores, authorship perceptions, and SBERT similarity to the textbook.
Problem
Whether LLM-assisted construction of multiple-choice assessment questions affects student outcomes and whether students can tell human-authored from LLM-authored questions.
Approach
A controlled classroom experiment that pairs each instructor MCQ with a ChatGPT-generated counterpart that passes a two-pass instructor validation, randomly assigns one of the pair to each student, and measures scores, perceived authorship, and textbook similarity via SBERT.
Key results
- Students scored about 9% lower on LLM-authored MCQs (0.74 vs 0.83, z=2.70, p<.01).
- Students could not distinguish human from LLM authorship (Mann-Whitney z=1.02, p=.31).
- LLM-authored questions were more similar to the course textbook than human ones (0.21 vs 0.14, z=-5.89, p<.001); conditional inference tree AUC 0.76 vs logistic regression 0.65.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use instructor-validated ChatGPT-4o-aided questions instead of human-authored questions | Operating systems course assessments · Operating systems course | score: .827 -> .739 | Hurt |
| Use instructor-validated ChatGPT-4o-aided questions instead of human-authored questions | Operating systems course assessments · Operating systems course | perception: .615 -> .598 | No clear effect |
| Use instructor-validated ChatGPT-4o-aided questions instead of human-authored questions | Operating systems course assessments · Operating systems course | cosine similarity to textbook: .144 -> .212 | Helped |
| Use a Conditional Inference Tree instead of logistic regression, modeling interactions between cosine similarity and score | Question authorship classification · Question authorship classification | AUC: 0.65 -> 0.76 | Helped |
Surprising findings
- Students could not tell whether questions were LLM-authored, yet scored lower on LLM-authored ones.
- Most LLM-generated MCQs failed the instructor's validation and were discarded.
Limitations
- Single instructor, single course (operating systems) with 25 students, so results may not generalize.
- Only SBERT and the single required textbook used for similarity; results depend on the specific LLM and its training data.
- Validation relied on one instructor's judgment; LLM prompts were zero-shot without iteration.
Experimental setup
Datasets: Modern Operating Systems (Tanenbaum) textbook instructor-written course MCQs
Metrics: mean score AUC Mann-Whitney U silhouette score cosine similarity
Baselines: instructor-authored questions
Builds on: Haladyna MCQ writing guidelines SBERT
Closest papers by meaning
- At Which Training Stage Does Code Data Help LLMs Reasoning? (ICLR 2024 · similarity 0.88)
- Students’ Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming (AAAI 2024 · similarity 0.87)
- MerryQuery: A Trustworthy LLM-Powered Tool Providing Personalized Support for Educators and Students (AAAI 2025 · similarity 0.87)
- How Teachers Can Use Large Language Models and Bloom’s Taxonomy to Create Educational Quizzes (AAAI 2024 · similarity 0.86)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.86)
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.