Research map / Label-efficient and robust learning / Recognition and long-tail
Practical Sentiment Analysis for Education: The Power of Student Crowdsourcing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that student crowdsourced sentiment labels (aggregated with weighted majority vote) can train supervised SVM and random forest models that predict sentiment of student unit-of-study reflections nearly as well as expert labels and better than self-reported labels.
Problem
Sentiment analysis of written student feedback (unit-of-study reflections) in large computer science courses lacks reliable, cheap training labels; dictionary methods are limited to major languages and lack context, LLMs raise bias and privacy concerns, and student self-reported labels are noisy.
Approach
Use student crowdsourced labels, aggregated via Dawid-Skene-style EM weighted majority voting, as training labels for supervised TF-IDF plus VADER-feature classifiers (RF and SVM) for education sentiment analysis, and frame the crowdsourcing as a hands-on course activity.
Key results
- Training RF on FL18+FL19 with crowdsourced labels raised negative-class recall from 64.5% (self-reported) to 81.1%, and combined CS+SR labels with more data reached 84.1% for RF.
- Crowdsourced labels agreed more with expert labels (weighted kappa 0.61) than self-reported labels did (0.53).
- Supervised RF and SVM models outperformed the VADER dictionary baseline on all reported measures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace self-reported labels with crowdsourced labels for training | RF and SVM sentiment classifiers · FL18+FL19 | negative class recall: 64.5 to 81.1 (RF); 19.8 to 70.1 (SVM) | Helped |
| Replace self-reported labels with crowdsourced labels for training | SVM sentiment classifier · FL18+FL19 | weighted recall: 62.6 to 73.7 | Helped |
| Replace self-reported labels with crowdsourced labels for training | RF sentiment classifier · FL18+FL19+REST | weighted recall: 65.5 to 63.7 | Hurt |
| Add REST training data labeled with student self-reports | SVM trained on FL18+FL19 SR labels · FL18+FL19+REST | negative class recall: 19.8 to 63.0 | Helped |
| Use CS+SR labels on FL18+FL19 and add REST with SR labels | RF and SVM sentiment classifiers · FL18+FL19+REST | negative class recall: 83.8 to 84.1 (RF); 79.5 to 80.2 (SVM) | Helped |
| Use crowdsourced labels rather than student self-reports as sentiment judgments | Sentiment labels for USRs · FL18 | weighted Cohen’s kappa with HE labels: 0.53 to 0.61 | Helped |
Surprising findings
- Self-reported star ratings were closer to random labels by RMSD (1.21) than expert labels were (1.26) relative to random, and often disagreed with the written text.
Limitations
- Human expert labels exist only for the FL18 dataset, limiting ground-truth evaluation.
- Student-written text may itself be dishonest or inaccurate, and response and sampling bias are not addressed.
- Only traditional ML models were tested; LLMs and multi-label learning were left for future work.
Experimental setup
Datasets: student unit-of-study reflections from Cloud Computing and Introduction to Computer Science courses (2016-2020)
Benchmarks: FL18 held-out human expert-annotated labels
Metrics: weighted recall weighted precision weighted F1-score negative class recall
Baselines: VADER self-reported labels training human expert-annotated labels training
Builds on: VADER Dawid-Skene EM Nasim et al. hybrid lexicon-ML approach
Closest papers by meaning
- Sentiment Analysis via Deep Hybrid Textual-Crowd Learning Model (AAAI 2018 · similarity 0.87)
- Advancing Multimodal Teacher Sentiment Analysis: The Large-Scale T-MED Dataset & the Effective AAM-TSA Model (AAAI 2026 · similarity 0.86)
- ROAR: Robust Label Ranking for Social Emotion Mining (AAAI 2018 · similarity 0.84)
- Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability (AAAI 2024 · similarity 0.83)
- Intelligent Calibration for Bias Reduction in Sentiment Corpora Annotation Process (AAAI 2024 · similarity 0.81)
- Unbiased Multi-Label Learning from Crowdsourced Annotations (ICML 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.