Research map / Large language models / AI and society
A Toolbox for Modelling Engagement with Educational Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the PEEKC dataset of AI-related educational video engagement and the TrueLearn Python library of online Bayesian learner-state models with visualisations, benchmarked against content-based and KT baselines.
Problem
Learner engagement modelling with educational videos lacks public in-the-wild datasets and accessible, interpretable, data-efficient online learner models; existing KT/IRT tools focus on test-taking rather than video consumption.
Approach
Releases a watch-time-labelled dataset with Wikipedia-based knowledge components per 5-minute fragment, and packages TrueLearn online Bayesian learner models (Interest, Novelty, INK) with open-learner visualisations in a scikit-learn-style library.
Key results
- TrueLearn INK achieves the highest accuracy (78.32%) and precision (64.32%) on PEEKC, versus 55-64% accuracy for baselines.
- TrueLearn Novelty achieves the best recall (80.91%) and F1 (65.53%), vs best baseline Jaccard-User F1 of 61.22.
- The PEEKC dataset contains 290,535 interaction events from 20,019 learners across 8,801 videos and 36,408 fragments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TrueLearn Interest instead of the strongest baseline, Jaccard_U | PEEKC engagement prediction · PEEKC test | F1: 61.22 -> 63.00 | Helped |
| Use TrueLearn Novelty instead of the strongest baseline, Jaccard_U | PEEKC engagement prediction · PEEKC test | F1: 61.22 -> 65.53 | Helped |
| Use TrueLearn INK instead of the strongest baseline, Jaccard_U | PEEKC engagement prediction · PEEKC test | F1: 61.22 -> 64.00 | Helped |
| Vary the number of knowledge components used by Cosine and Jaccard content representations (ablation) | Cosine and Jaccard baselines · PEEKC test | F1 | No clear effect |
Surprising findings
- The KT baseline performs worst on F1 (34.51) with recall of only 28.56%, despite being a learner model.
Limitations
- Exclusively supports online learning algorithms; no batch or deep learning models.
- Not compatible with some scikit-learn helpers such as grid search.
- Visualisation effectiveness not yet validated through user studies.
Experimental setup
Datasets: PEEKC
Benchmarks: PEEKC
Metrics: accuracy precision recall F1
Baselines: Cosine Jaccard (concept) Jaccard (user) TF (Binary) TF (Cosine) Knowledge Tracing (KT)
Builds on: TrueLearn Wikification scikit-learn pyBKT Knowledge Tracing
Closest papers by meaning
- An XAI Social Media Platform for Teaching K-12 Students AI-Driven Profiling, Clustering, and Engagement-Based Recommending (AAAI 2025 · similarity 0.87)
- From Raw Video to Pedagogical Insights: A Unified Framework for Student Behavior Analysis (AAAI 2024 · similarity 0.85)
- Learning About Algorithm Auditing in Five Steps: Scaffolding How High School Youth Can Systematically and Critically Evaluate Machine Learning Applications (AAAI 2025 · similarity 0.83)
- Word2Vec4Kids: Interactive Challenges to Introduce Middle School Students to Word Embeddings (AAAI 2025 · similarity 0.82)
- Build Your Own Robot Friend: An Open-Source Learning Module for Accessible and Engaging AI Education (AAAI 2024 · similarity 0.82)
- Shaping AI Interest in Rural Middle Schools with Unplugged Learning: Gender Differences and Teacher Insights (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.