Research map / Video / Video-language understanding
From Raw Video to Pedagogical Insights: A Unified Framework for Student Behavior Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an end-to-end pipeline that runs skeleton-based temporal action detection (MotionBERT + ActionFormer-style transformer) on classroom video and feeds the detected action timelines to an LLM (GPT-4 and others) to generate pedagogical reports.
Problem
Automated student behavior analysis in classrooms is limited to single or multi-class action detection, lacks generalizability, raises privacy concerns with RGB video, and does not turn analytics into actionable insights for educators.
Approach
Frames student behavior analysis as temporal action detection on privacy-preserving 2D skeleton sequences, built from synthetic long sequences of NTU RGB+D actions with multi-view synthesis and spline motion interpolation, and couples the detections with an LLM prompted with educational identity and theory to write reports.
Key results
- Full TAD pipeline (MotionBERT + Transformer, multi-view, interpolation, consistency) reaches 77.89 average mAP on the test set, versus 27.22 for ST-GCN + Conv.
- Multi-view synthesis gives the largest gain in ablation, raising average mAP from 20.35 to 28.99 (consistency and interpolation add further gains to 44.62 and 68.77).
- In a 15-evaluator user study, the combined identity and theoretical-guidance prompt scored 7.91 out of 10 versus 4.25 for the basic prompt; GPT-4 was rated slightly above GPT-3.5 and both above LLaMA-7B/13B in GPT-4-based automatic evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add splicing consistency, motion interpolation, and multi-view synthesis (ablation) | Skeleton-based TAD · Constructed dataset test set | Average mAP: 20.35 -> 77.89 | Helped |
| Add motion interpolation and multi-view synthesis to splicing consistency (ablation) | Skeleton-based TAD · Constructed dataset test set | Average mAP: 28.99 -> 77.89 | Helped |
| Add multi-view synthesis to consistency and motion interpolation (ablation) | Skeleton-based TAD · Constructed dataset test set | Average mAP: 44.62 -> 77.89 | Helped |
| Add motion interpolation to consistency and multi-view synthesis (ablation) | Skeleton-based TAD · Constructed dataset test set | Average mAP: 68.77 -> 77.89 | Helped |
| Increase training data size from 500 to 2000 sequences (ablation) | Skeleton-based TAD · Constructed dataset test set | Average mAP: 25.03 -> 75.36 | Helped |
| Use MotionBert + Transformer instead of ST-GCN + Conv (ablation) | Skeleton-based TAD · Constructed dataset test set | Average mAP: 27.22 -> 77.89 | Helped |
| Replace ST-GCN with MotionBert while retaining Transformer (ablation) | Skeleton-based TAD · Constructed dataset test set | Average mAP: 44.06 -> 77.89 | Helped |
| Replace Conv with Transformer while retaining MotionBert (ablation) | Skeleton-based TAD · Constructed dataset test set | Average mAP: 50.10 -> 77.89 | Helped |
Surprising findings
- Adding a theory-based context prompt alone (Contextual Analysis) lowered the user-study score from 4.25 to 5.08 only marginally, while Recommendations raised it to 7.08, showing prompt-dimension effects vary.
Limitations
- Training data is synthetic from NTU RGB+D and not from real classroom settings, which may cause performance drops in practice.
- Only individual actions are handled; group or interactive activities such as discussions are not captured.
- Report quality evaluation relies on a small user study and GPT-4-based automatic scoring rather than validated educational outcomes.
Experimental setup
Datasets: NTU RGB+D custom synthetic classroom TAD dataset
Benchmarks: custom synthetic classroom TAD test set
Metrics: mAP at tIoU 0.3/0.5/0.7 average mAP user-study score (out of 10) GPT-4-based report rating
Baselines: ST-GCN + Conv ST-GCN + Transformer MotionBERT + Conv LLaMA-7B LLaMA-13B GPT-3.5
Builds on: ActionFormer MotionBERT NTU RGB+D AlphaPose ST-GCN
Closest papers by meaning
- A Toolbox for Modelling Engagement with Educational Videos (AAAI 2024 · similarity 0.85)
- Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement (AAAI 2025 · similarity 0.83)
- Watch and Learn: Learning to Use Computers from Online Videos (CVPR 2026 · similarity 0.83)
- BEAGLE: Behavior-Enforced Agent for Grounded Learner Emulation (NeurIPS 2026 · similarity 0.83)
- Conversational Learning Diagnosis via Reasoning Multi-Turn Interactive Learning (AAAI 2026 · similarity 0.82)
- Opening the Black Box: Unraveling the Classroom Dialogue Analysis (Student Abstract) (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.