Research map / Large language models / AI and society
Students’ Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Surveyed 58 CS1 Java students on ChatGPT API-generated feedback for four lab assignments, comparing prompts with and without student code and assessing alignment with Shute's formative feedback guidelines.
Problem
Students' subjective perceptions of LLM-generated feedback on introductory programming assignments are under-explored, including whether it is formative, whether including student code in the prompt helps, and what improvements students want.
Approach
Generate ChatGPT feedback for real CS1 lab submissions in two variants (with and without student code in the user prompt), distribute them after grades are released, and measure student perceptions against Shute's formative feedback guidelines using surveys and thematic analysis.
Key results
- About 98% (100/102) of responses agreed feedback was unbiased and objective, and 95% (97/102) agreed it was presented in manageable units.
- 72.5% (74/102) preferred feedback generated with student code in the prompt over the version without code.
- Only 40% (41/102) rated the feedback as clear enough on the specific and clear guideline, indicating room for improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Include students' code in the prompt rather than exclude it | ChatGPT-generated Java programming feedback · Four Java labs | student preference: 72.5% (74/102) preferred feedback with code | Helped |
| Include students' code in the prompt rather than exclude it | ChatGPT-generated Java programming feedback · Four Java labs | student preference by lab: 70.37% and 75.00% | Helped |
| Generate feedback with students' code | ChatGPT-generated Java programming feedback · Four Java labs | perceived specificity, clarity, and corrective suggestions: specificity (33/74), clarity (15/74) and more corrective suggestions (9/74) | Helped |
| Generate feedback without students' code | ChatGPT-generated Java programming feedback · Four Java labs | student preference: 20% | Mixed |
Surprising findings
- About 20% of students preferred the feedback without code, mainly for its specificity and motivational support.
- Students showed diverging preferences on feedback tone, with some wanting encouragement and others wanting critical feedback.
Limitations
- Small sample of 58 students and four labs limits generalizability.
- Only student perceptions were measured, with no learning outcome analysis.
- Formative alignment judged by an arbitrary 70% threshold without a baseline.
Experimental setup
Datasets: student Java lab submissions from a CS1 course failed test cases from instructor test scripts GitHub autograder feedback
Metrics: percentage of favorable survey responses feedback preference share thematic code counts
Baselines: feedback without student code in prompt
Builds on: Shute formative feedback guidelines
Closest papers by meaning
- LLMs in the Classroom: Outcomes and Perceptions of Questions Written with the Aid of AI (AAAI 2025 · similarity 0.87)
- Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation (NeurIPS 2024 · similarity 0.87)
- Understanding GenAI for Teaching and Learning in Secondary Classrooms (AAAI 2025 · similarity 0.86)
- ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments (ICLR 2025 · similarity 0.85)
- Automated Assessment of Student Self-Explanation in Code Comprehension Using Pre-Trained Language Models (AAAI 2025 · similarity 0.85)
- Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.