Research map / Large language models / Language, safety and interpretability
Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates nine prompt-specific and cross-prompt automated essay scoring methods on the PERSUADE 2.0 corpus for accuracy (QWK, MAE, PCC) and demographic fairness (OSA, OSD, CSD, MAED), finding prompt-specific models are more accurate but more economically biased, and simple SVMs are competitive cross-prompt.
Problem
Whether AES models trade off predictive accuracy, fairness across demographic groups, and generalizability across prompts, which has not been jointly evaluated for prompt-specific versus cross-prompt AES.
Approach
Systematically benchmarks nine representative AES methods (five prompt-specific, four cross-prompt) on a demographic-annotated public corpus, using seven metrics that jointly assess accuracy and group fairness, and compares the two settings directly.
Key results
- Prompt-specific models outperform cross-prompt models on average by 25.61% in QWK, 23.43% MAE reduction, and 18.06% PCC.
- Best prompt-specific BERT (3 Layers) reaches average QWK 0.811, MAE 0.440, PCC 0.817; best cross-prompt SVM (Reduced) average QWK 0.744.
- Prompt-specific models show more economic-status bias (average prompts flagged: 2.4 vs 1.25 in CSD); SVM (Full) has the best fairness with MAED -0.006.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use prompt-specific rather than cross-prompt AES models | AES models · PERSUADE 2.0, 12 prompts | QWK: +25.61% | Helped |
| Use prompt-specific rather than cross-prompt AES models | AES models · PERSUADE 2.0, 12 prompts | MAE: 23.43% reduction | Helped |
| Use prompt-specific rather than cross-prompt AES models | AES models · PERSUADE 2.0, 12 prompts | PCC: +18.06% | Helped |
| Use BERT (3 Layers) instead of SVM (Reduced) | AES scoring · PERSUADE 2.0, 12 prompts | QWK: 9.00% | Helped |
| Use BERT (3 Layers) instead of SVM (Reduced) | AES scoring · PERSUADE 2.0, 12 prompts | MAE: 8.71% | Helped |
| Use BERT (3 Layers) instead of SVM (Reduced) | AES scoring · PERSUADE 2.0, 12 prompts | PCC: 5.42% | Helped |
| Use deep neural networks rather than traditional machine learning | Prompt-specific AES · PERSUADE 2.0, 12 prompts | QWK: 0.811 (QWK) | Helped |
| Use SVM (Reduced) rather than deep learning methods | Cross-prompt AES · PERSUADE 2.0, 12 prompts | QWK: 0.744 | Helped |
| Use prompt-specific rather than cross-prompt models | AES fairness across economic-status groups · PERSUADE 2.0, economic status | Average number of prompts with OSA bias: 2.0 vs. 0.5 | Mixed |
| Use prompt-specific rather than cross-prompt models | AES fairness across economic-status groups · PERSUADE 2.0, economic status | Average number of prompts with OSD bias: 2.4 vs. 2.25 | Mixed |
| Use prompt-specific rather than cross-prompt models | AES fairness across economic-status groups · PERSUADE 2.0, economic status | Average number of prompts with CSD bias: 2.4 vs. 1.25 | Hurt |
| Use prompt-specific rather than cross-prompt models | AES fairness across economic-status groups · PERSUADE 2.0, economic status | Absolute MAED: 2.63% | Hurt |
Surprising findings
- Simple SVM with handcrafted features in the cross-prompt setting outperforms deep neural cross-prompt models such as PAES and TDNN.
- Prompt-specific models, though more accurate, exhibit more economic-status bias than cross-prompt models.
Limitations
- Experiments restricted to a single dataset (PERSUADE 2.0).
- Analysis focuses on evaluating fairness without proposing mitigation methods.
Experimental setup
Datasets: PERSUADE 2.0 corpus
Benchmarks: PERSUADE 2.0 corpus
Metrics: QWK MAE PCC OSA OSD CSD MAED
Baselines: SVM (Full) SKIPFLOW-LSTM CNN-LSTM-ATT R$^2$BERT BERT (3 Layers) SVM (Reduced) RankSVM PAES TDNN
Builds on: BERT SKIPFLOW-LSTM R$^2$BERT RankSVM TDNN
Closest papers by meaning
- KAES: Multi-aspect Shared Knowledge Finding and Aligning for Cross-prompt Automated Scoring of Essay Traits (AAAI 2025 · similarity 0.89)
- Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Education (AAAI 2024 · similarity 0.85)
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.84)
- From Consumers to Critical Users: Prompty, an AI Literacy Tool for High School Students (AAAI 2024 · similarity 0.84)
- IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation (NeurIPS 2024 · similarity 0.83)
- Practical Sentiment Analysis for Education: The Power of Student Crowdsourcing (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.