Research map / Large language models / AI and society
iLLuMinaTE: An LLM-XAI Framework Leveraging Social Science Explanation Theories Towards Actionable Student Performance Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes iLLuMinaTE, a zero-shot chain-of-prompts LLM pipeline that turns post-hoc XAI outputs (LIME, CEM, MC-LIME) for student performance models into theory-grounded, actionable feedback, evaluated by expert/GPT-4o annotation, readability metrics, and a 114-student preference study.
Problem
Post-hoc XAI explanations of student performance models are hard for non-technical educators and students to understand and act on.
Approach
Use LLMs as communicators of XAI outputs via a three-stage chain of prompts (causal connection, explanation selection grounded in eight social science explanation theories, and explanation presentation following Hattie's feedback framework and Grice's maxims), with no fine-tuning.
Key results
- Students preferred iLLuMinaTE explanations over baseline LIME/CEM/MC-LIME explanations 89.52% of the time (Kruskal-Wallis H=176.38, p<.0001).
- Explanation selection alignment scores are high (overall >0.82 for all explainers and theories; e.g., RaR+Contrastive 0.985 with CEM); GPT-4o vs human annotation agreement 94.68%.
- Simulated actions chosen by students improved performance by 13.5% (LIME), 14.2% (CEM), and 20.7% (MC-LIME) for students preferring iLLuMinaTE explanations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Present post-hoc explanations as iLLuMinaTE theory-driven text and visual feedback rather than the traditional explainer output | LIME, CEM, and MC-LIME explanations · Student explanation preference study | preference rate: 89.52% preferred iLLuMinaTE | Helped |
| Use iLLuMinaTE feedback to guide student-selected actions instead of baseline explanation-based choices | Student performance simulation · LIME, CEM, and MC-LIME interventions | average performance improvement: 13.5% for LIME, 14.2% for CEM, and 20.7% for MC-LIME | Helped |
| Use Contrastive theory prompting rather than other theory prompts (ablation) | iLLuMinaTE · Student performance simulation | average performance improvement: 28.2% | Helped |
| Use Necessity Robustness theory prompting rather than other theory prompts (ablation) | iLLuMinaTE · Student performance simulation | average performance improvement: 24.9% | Helped |
| Use MC-LIME instead of LIME or CEM as the underlying explainer (ablation) | iLLuMinaTE action-generation pipeline · Student performance simulation | average performance improvement: 13.5% for LIME, 14.2% for CEM, and 20.7% for MC-LIME | Helped |
| Use GPT-4o rather than Gemma2 9b or Llama3 70b for explanation selection (ablation) | iLLuMinaTE · Explanation Selection | instruction-following score: GPT-4o explanations scored the highest, closely followed by Gemma2 9b, and Llama3 70b; all of the 95% CI overlap. | Mixed |
| Use Llama3 70b rather than GPT-4o or Gemma2 9b for explanation presentation (ablation) | iLLuMinaTE · Explanation Presentation | instruction-following score | Helped |
| Replace baseline explanation text with theory-based iLLuMinaTE explanations | Student explanation ratings · Student ratings | Usefulness: 4.18 vs 3.59 | Helped |
| Vary the social science theory prompt across explainers (ablation) | iLLuMinaTE explanation selection · Theory alignment evaluation | overall alignment | No clear effect |
| Vary the underlying explainer among LIME, CEM, and MC-LIME (ablation) | iLLuMinaTE explanation presentation · Readability metrics across courses | readability and grammar | No clear effect |
| Generate the explanation with theory-based prompts rather than CoT prompting (ablation) | iLLuMinaTE · Student ratings and actionability | usefulness and actionability | Mixed |
Surprising findings
- Llama3 70B scored highest on explanation presentation while GPT-4o was best at explanation selection, suggesting smaller open models may suffice for presentation.
- Q4 (using course content and structure) was consistently the weakest criterion, with some theories scoring below 65% (e.g., Statistical Relevance 0.453 with CEM).
Limitations
- Evaluation of explanations is hard to separate from students' prior beliefs and knowledge.
- Explainer variability; no significant differences between explainers observed in alignment scores.
- Actionability measured via simulation on a BiLSTM rather than real learning gains.
Experimental setup
Datasets: clickstream data from three MOOCs (DSP, Geo, VA)
Benchmarks: iLLuMinaTE explanation alignment rubric (decomposed yes/no questions) Flesch-Kincaid Gunning Fog SMOG LanguageTool grammar
Metrics: ratio of yes answers readability grade scores student preference percentage Likert ratings (usefulness, trustworthiness, actionability, simulated performance improvement Cohen's kappa
Baselines: LIME CEM MC-LIME Chain-of-Thought
Builds on: LIME Contrastive Explanation Method MC-LIME Chain-of-Thought prompting Rephrase and Respond
Closest papers by meaning
- Comparing Explanations is not Enough,Explain the Change: New Standards are Needed to Explain Behavioral Shifts in Large Language Models (NeurIPS 2026 · similarity 0.86)
- Navigating the Maze of Explainable AI: A Systematic Approach to Evaluating Methods and Metrics (NeurIPS 2024 · similarity 0.86)
- EduMirror: Modeling Educational Social Dynamics with Value-driven Multi-agent Simulation (ICML 2026 · similarity 0.86)
- Position: LLM Social Simulations Are a Promising Research Method (ICML 2025 · similarity 0.85)
- Explain with Visual Keypoints Like a Real Mentor! A Benchmark for Multimodal Solution Explanation (AAAI 2026 · similarity 0.85)
- K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.