Research map / Label-efficient and robust learning / Recognition and long-tail
KAES: Multi-aspect Shared Knowledge Finding and Aligning for Cross-prompt Automated Scoring of Essay Traits
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KAES, which iteratively clusters encoder representations of source-prompt essays to find multi-aspect shared knowledge, aligns essays to clusters via a KL clustering loss, and uses GPT-3.5 descriptions encoded by BERT as explicit knowledge features for cross-prompt multi-trait essay scoring on ASAP/ASAP++.
Problem
Cross-prompt automated essay scoring must learn shared knowledge across prompts, but prior methods capture only coarse, generic shared knowledge rather than detailed multi-aspect knowledge; the acquired knowledge is also implicit and hard to interpret.
Approach
Alternate multi-trait scoring training with a knowledge finding and aligning step: cluster stratified-sampled essays from all source prompts, align them to cluster centers with a DEC-style KL loss, describe each cluster's representative essays with an LLM, encode the description with BERT, and concatenate the nearest-centroid knowledge representation to the essay representation for scoring.
Key results
- KAES(hier)+gmm averages 0.597 QWK over prompts versus 0.575 for PLAES, a 2.2% average improvement, with a 5.1% gain on P5.
- KAES(hier)+kmeans with 837K parameters reaches 0.595 prompt QWK versus 0.575 for PLAES (1.47M parameters).
- Ablation: removing both clustering and LLM knowledge drops prompt QWK from 0.595 to 0.559.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use KAES(hier)+gmm instead of PLAES | Cross-prompt AES on ASAP++ · ASAP++ prompts (AVG) | QWK: 0.575 -> 0.597 | Helped |
| use KAES(hier)+gmm instead of PLAES | Cross-prompt AES on ASAP++ · ASAP++ traits (AVG) | QWK: 0.570 -> 0.594 | Helped |
| add LLM-based knowledge capture (ablation) | KAES · ASAP++ prompts (AVG) | QWK: 0.586 -> 0.595 | Helped |
| add LLM-based knowledge capture (ablation) | KAES · ASAP++ traits (AVG) | QWK: 0.581 -> 0.592 | Helped |
| add cluster-based alignment and LLM-based knowledge capture (ablation) | KAES · ASAP++ prompts (AVG) | QWK: 0.559 -> 0.595 | Helped |
| add cluster-based alignment and LLM-based knowledge capture (ablation) | KAES · ASAP++ traits (AVG) | QWK: 0.553 -> 0.592 | Helped |
| add cluster-based alignment (ablation) | Hierarchical CNN-LSTM · ASAP++ in-domain (AVG source) | QWK: 0.737 -> 0.729 | Hurt |
| add cluster-based alignment and LLM-based knowledge capture (ablation) | Hierarchical CNN-LSTM · ASAP++ in-domain (AVG source) | QWK: 0.737 -> 0.733 | Hurt |
| add cluster-based alignment (ablation) | Hierarchical CNN-LSTM · ASAP++ P1 (target) | QWK: 0.562 -> 0.634 | Helped |
| add cluster-based alignment and LLM-based knowledge capture (ablation) | Hierarchical CNN-LSTM · ASAP++ P1 (target) | QWK: 0.562 -> 0.675 | Helped |
| increase the number of sampled essays per prompt (ablation) | KAES · ASAP++ | QWK | Helped |
| use 50 sampled essays per prompt (ablation) | KAES · ASAP++ | QWK | Hurt |
Surprising findings
- Cross-prompt performance improved while in-domain performance stayed roughly stable or slightly decreased, per the withheld-essay experiment on P1 and P2.
- The hierarchical encoder outperformed the BERT encoder by about 7% on prompts, which the authors note for the variants.
Limitations
- Adds clustering time cost and external GPU memory use (peak 17,284 MiB at 10% sampling).
- Relies on GPT-3.5 for explicit knowledge descriptions, adding API cost and dependence on a proprietary model.
- Kc and sampling size are sensitive hyperparameters; results depend on cluster count choices.
Experimental setup
Datasets: ASAP ASAP++
Benchmarks: ASAP ASAP++
Metrics: QWK
Baselines: GPT-3.5-turbo (0-shot) Finetune BERT Hi att AES aug PAES CTS no att CTS PMAES PLAES
Builds on: Ridley et al. 2021 (CTS) Chen and Li 2024 (PLAES) Chen and Li 2023 (PMAES) Wang et al. 2019 (deep embedded clustering-style distribution alignment)
Closest papers by meaning
- Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability (AAAI 2024 · similarity 0.89)
- Towards Automatic Boundary Detection for Human-AI Collaborative Hybrid Essay in Education (AAAI 2024 · similarity 0.85)
- CriSPO: Multi-Aspect Critique-Suggestion-guided Automatic Prompt Optimization for Text Generation (AAAI 2025 · similarity 0.83)
- A Systematic Exploration of Knowledge Graph Alignment with Large Language Models in Retrieval Augmented Generation (AAAI 2025 · similarity 0.82)
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.82)
- Knowledge Tagging with Large Language Model Based Multi-Agent System (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.