Research map / Large language models / Language, safety and interpretability
Automated Assessment of Student Self-Explanation in Code Comprehension Using Pre-Trained Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes BERT, RoBERTa, DistilBERT, CodeBERT and SciBERT as regression models to score students' line-by-line self-explanations of Java code against expert explanations, comparing to TF-IDF, BERTScore and USE baselines.
Problem
Automatically assessing students' free-text self-explanations of source code during code comprehension tasks, where explanations mix natural language with code references and existing work relies on semantic similarity or traditional ML features.
Approach
Applies fine-tuned transformer encoders with a regression head to predict semantic similarity between student and expert line-level explanations, and compares three aggregation schemes (max-of-max, max-of-avg, avg-of-max) for combining multi-sentence pairs.
Key results
- SciBERT achieved the best Pearson correlation of 0.784 and Spearman 0.775 under avg-of-max aggregation.
- All fine-tuned transformers outperformed TF-IDF, BERTScore and USE baselines (best baseline BERTScore Pearson 0.397 under max-of-avg).
- CodeBERT reached Pearson 0.739 under max-of-avg, marginally above SciBERT's 0.736.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune pre-trained transformer models rather than use the semantic-similarity baselines | Student code explanation assessment · Code explanation assessment | Pearson and Spearman correlation | Helped |
| Use SciBERT rather than CodeBERT | Student code explanation assessment · Max of Max and Avg of Max | Pearson correlation: 0.733 vs 0.656; 0.784 vs 0.734 | Helped |
| Use CodeBERT rather than SciBERT | Student code explanation assessment · Max of Avg | Pearson and Spearman correlation: 0.739 vs 0.736; 0.739 vs 0.686 | Helped |
| Use Avg of Max aggregation rather than Max of Max or Max of Avg | Fine-tuned transformer models · Code explanation assessment | Pearson and Spearman correlation | Helped |
Surprising findings
- CodeBERT, a code-oriented model, did not outperform SciBERT overall, lagging it under max-of-max and avg-of-max aggregation.
Limitations
- Relies on a single expert reference explanation per line, which may not capture valid variation in student responses.
- Struggles with alternative but correct phrasings and longer multi-sentence explanations.
- Does not incorporate program goal or other context; generalizability to other languages/domains untested.
Experimental setup
Datasets: SelfCode 2.0
Benchmarks: SelfCode 2.0
Metrics: Pearson correlation Spearman correlation
Baselines: TF-IDF BERTScore Universal Sentence Encoder
Builds on: BERT RoBERTa DistilBERT CodeBERT SciBERT
Closest papers by meaning
- LLMs in the Classroom: Outcomes and Perceptions of Questions Written with the Aid of AI (AAAI 2025 · similarity 0.85)
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.85)
- Students’ Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming (AAAI 2024 · similarity 0.85)
- Detecting AI-Generated Code Assignments Using Perplexity of Large Language Models (AAAI 2024 · similarity 0.85)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.85)
- Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.