Research map / Reinforcement learning / Visual and representation RL
Get a Head Start: On-Demand Pedagogical Policy Selection in Intelligent Tutoring
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EDUPLANNER, an on-demand pedagogical policy selector that partitions students into subgroups from initial logs, augments data with a VAE-in-MDP, and picks the best of a few DQN/expert policies per arriving student in a Probability ITS.
Problem
Off-policy evaluation and selection for intelligent tutoring systems is hard because student subgroups favor different pedagogical policies, and policies cannot be retrained or evaluated extensively due to expert sanity checks and deployment cost.
Approach
Formulates on-demand pedagogical policy selection (ODPS): partition the initial-state space into subgroups using TICC clustering over distilled student-sensitive features chosen by a data-driven feature taxonomy, enrich each subgroup with VAE-in-MDP augmented trajectories, and deploy the subgroup's best OPE-estimated policy to each arriving student.
Key results
- EDUPLANNER yields the highest average normalized learning gain among compared selection methods in the real classroom test on the Probability ITS.
- For low-performing subgroups K3 and K4, EDUPLANNER achieved average NLGs of 24 and 32 versus -12 and -20 for the best OPE-based selections.
- EDUPLANNER produces the smallest gap between estimated and true policy returns, similar to OPE AugRRS (FQE).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EDUPLANNER instead of the best OPE-related methods | Policy selection for Probability ITS low-performing subgroup K3 · Probability ITS, K3, testing semester | average NLG: -12 -> 24 | Helped |
| Use EDUPLANNER instead of the best OPE-related methods | Policy selection for Probability ITS low-performing subgroup K4 · Probability ITS, K4, testing semester | average NLG: -20 -> 32 | Helped |
| Use all recorded features for subgroup partitioning instead of feature taxonomy distillation (ablation) | EDUPLANNER · Probability ITS | student performance | Hurt |
| Remove sample enrichment with VAE-augmented samples (ablation) | EDUPLANNER · Probability ITS | student performance | Hurt |
| Select one policy across all subgroups instead of selecting policies per subgroup (ablation) | EDUPLANNER · Probability ITS | student performance | Hurt |
| Apply OPE with repeated random sampling (ablation) | OPE policy selection · Probability ITS | policy-selection performance | No clear effect |
| Apply OPE with repeated random sampling on VAE-augmented samples (ablation) | OPE policy selection · Probability ITS | policy-selection performance | No clear effect |
| Use augmented samples with repeated random sampling (ablation) | OPE estimators · Probability ITS | estimation error | Helped |
| Partition subgroups using longer trajectory information (ablation) | OPE AugRRS · Probability ITS | mean absolute error | Helped |
| Partition subgroups using longer trajectory information (ablation) | OPE AugRRS · Probability ITS | policy-selection performance | No clear effect |
Surprising findings
- OPE-based selection tends to pick sub-optimal policies that nonetheless outperform the expert policy on average, and augmentation with RRS did not fix the selection problem despite improved estimation accuracy.
- Sub-optimal OPE-selected policies had negative effects on low-performing subgroups even though they were positive overall.
Limitations
- Assumes initial logs are informative enough for subgroup partitioning; cold-start settings are not addressed.
- Evaluated on a single Probability ITS course with small subgroup sizes (e.g., K4 has 6 test students).
Experimental setup
Datasets: Probability ITS historical student logs (459k logs, 1,148 students)
Benchmarks: Probability ITS classroom study (140 students, testing semester)
Metrics: normalized learning gain distance between estimated and true policy return
Baselines: WIS PDIS FQE WDR MAGIC DualDICE OPE RRS OPE AugRRS expert policy
Builds on: TICC Fitted Q Evaluation MAGIC DualDICE Repeated random sampling OPE (Nie et al. 2022)
Closest papers by meaning
- Off-Policy Selection for Initiating Human-Centric Experimental Design (NeurIPS 2024 · similarity 0.88)
- The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.87)
- OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators (NeurIPS 2024 · similarity 0.86)
- SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervision (NeurIPS 2026 · similarity 0.86)
- Online Reinforcement Learning-Based Pedagogical Planning for Narrative-Centered Learning Environments (AAAI 2024 · similarity 0.86)
- Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.