Research map / Multi-agent systems and games / LLM agents

Scaling Equitable Reflection Assessment in Education via Large Language Models and Role-Based Feedback Agents

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a five-agent GPT-4o pipeline that scores adult learners' reflections against a four-dimension rubric and writes bias-aware, metacognitive feedback, evaluated against three human graders on scoring error, fairness gaps, feedback ratings, and cost.

Problem

Formative feedback on student reflections is hard to deliver equitably at scale in large or low-resource courses, and LLM graders may be inaccurate or biased for lower-scoring learners.

Approach

A role-based multi-agent workflow (Evaluator, Equity Monitor, Metacognitive Coach, Aggregator, Reflexion Reviewer) built on zero-shot GPT-4o prompting that produces structured rubric scores and short feedback, plus a band-based fairness metric comparing MAE across low- and high-ability learners.

Key results

• Overall rubric MAE of 0.467 and mean QWK of 0.459 between the system and human majority vote on 84 reflections.
• Human-AI ICC(2,1) of 0.41 versus 0.57 for human-human; worst-band fairness gap Delta_MAE of 0.50 with low-ability MAE 0.896 vs high-ability 0.396.
• Feedback quality Q(g)=3.967 on a 5-point scale, just below the 4.0 target; empathy highest at 4.223; cost about $0.00155 per reflection with gpt-4o-mini.

Surprising findings

• Human-AI agreement is lower than human-human agreement, and the model's error is substantially higher for low-ability learners (Delta_MAE 0.50), contrary to the paper's framing of equitable accuracy.
• The paper's abstract claims scores 'approach expert-level agreement', but the reported QWK (0.459) and ICC (0.41) are moderate and below human-human reliability.

Limitations

• Evaluation limited to 28 adult English-language learners; low-proficiency writers show higher error.
• Human-AI reliability is moderate (ICC 0.41) and feedback usefulness falls slightly below the 4.0 target.
• Model drift may affect performance over time; no causal learning-gain measurement.

Experimental setup

Datasets: 336 reflections from 28 adult learners in an AI literacy program (84 sampled reflections graded)

Benchmarks: in-house AI literacy reflection dataset

Metrics: MAE QWK ICC(2,1) worst-band MAE gap Likert feedback quality latency cost

Builds on: AutoGen Reflexion

Closest papers by meaning

• Agent4Edu: Generating Learner Response Data by Generative Agents for Intelligent Education Systems (AAAI 2025 · similarity 0.85)
• MerryQuery: A Trustworthy LLM-Powered Tool Providing Personalized Support for Educators and Students (AAAI 2025 · similarity 0.85)
• Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators (NeurIPS 2026 · similarity 0.85)
• From Consumers to Critical Users: Prompty, an AI Literacy Tool for High School Students (AAAI 2024 · similarity 0.84)
• DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback (ICLR 2025 · similarity 0.84)
• Automated Assessment of Student Self-Explanation in Code Comprehension Using Pre-Trained Language Models (AAAI 2025 · similarity 0.84)