Research map / Reinforcement learning / Offline RL
Online Reinforcement Learning-Based Pedagogical Planning for Narrative-Centered Learning Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes INSIGHT, an online RL framework (simulator, rule-based simulated student, double-DQN planner) for pedagogical planning in a CRYSTAL ISLAND-based narrative learning testbed, showing online DQN beats offline DQN/BCQ/CQL trained on synthetic data.
Problem
RL-based pedagogical planning in narrative-centered learning environments suffers from scarce training data and relies on offline RL, which cannot explore or evaluate on demand and yields suboptimal policies.
Approach
Couple a deterministic narrative learning environment simulator and a rule-based simulated student with an online double-DQN planner that gets immediate rewards from in-game score, enabling on-demand synthetic data generation, exploration, and direct policy evaluation instead of OPE.
Key results
- Online double DQN reaches mean reward 636 (SD=72), near the hand-designed optimal policy (696), converging after about 150,000 steps.
- Best offline results with 5,000 episodes: offline DQN 498 (SD=53), offline BCQ 510 (SD=51), offline CQL 444 (SD=96), all below online DQN.
- With only 100 offline episodes, offline DQN achieves mean reward 150, offline BCQ 139, and offline CQL 87.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use online DQN instead of offline DQN trained with 5,000 episodes | INSIGHT pedagogical planner · CRYSTAL ISLAND testbed, simulated evaluation | mean reward: 498 (SD=53) -> 636 (SD=72) | Helped |
| Use online DQN instead of offline BCQ trained with 5,000 episodes | INSIGHT pedagogical planner · CRYSTAL ISLAND testbed, simulated evaluation | mean reward: 510 (SD=51) -> 636 (SD=72) | Helped |
| Use online DQN instead of offline CQL trained with 5,000 episodes | INSIGHT pedagogical planner · CRYSTAL ISLAND testbed, simulated evaluation | mean reward: 444 (SD=96) -> 636 (SD=72) | Helped |
| Increase offline DQN training data from 100 to 500 episodes | Offline DQN · CRYSTAL ISLAND testbed, simulated evaluation | mean reward: 150 (SD=109) -> 392 (SD=116) | Helped |
| Increase offline DQN training data from 500 to 5,000 episodes | Offline DQN · CRYSTAL ISLAND testbed, simulated evaluation | mean reward: 392 (SD=116) -> 498 (SD=53) | Helped |
| Increase offline BCQ training data from 100 to 500 episodes | Offline BCQ · CRYSTAL ISLAND testbed, simulated evaluation | mean reward: 139 (SD=142) -> 373 (SD=127) | Helped |
| Increase offline BCQ training data from 500 to 5,000 episodes | Offline BCQ · CRYSTAL ISLAND testbed, simulated evaluation | mean reward: 373 (SD=127) -> 510 (SD=51) | Helped |
| Increase offline CQL training data from 100 to 1,000 episodes | Offline CQL · CRYSTAL ISLAND testbed, simulated evaluation | mean reward: 87 (SD=118) -> 405 (SD=62) | Helped |
| Increase offline CQL training data from 1,000 to 5,000 episodes | Offline CQL · CRYSTAL ISLAND testbed, simulated evaluation | mean reward: 405 (SD=62) -> 444 (SD=96) | Helped |
Surprising findings
- Offline BCQ with 5,000 episodes slightly exceeded offline DQN, and offline DQN matched CQL and BCQ at large data sizes, suggesting extrapolation error matters less with enough data.
Limitations
- Rule-based simulated student may not reflect real human behavior.
- Simulator abstracts low-level game mechanics, possibly with unexpected effects.
- In-game score rewards have not been validated for predicting learning in classroom deployment with a run-time RL planner.
Experimental setup
Datasets: synthetic data generated by rule-based simulated student in INSIGHT simulator
Benchmarks: INSIGHT simulator (CRYSTAL ISLAND-based testbed)
Metrics: mean total reward per student convergence steps standard deviation across 10 seeds
Baselines: offline DQN offline BCQ offline CQL hand-designed optimal policy random policy always alter policy always do not alter policy
Builds on: double DQN CRYSTAL ISLAND d3rlpy BCQ CQL
Closest papers by meaning
- Get a Head Start: On-Demand Pedagogical Policy Selection in Intelligent Tutoring (AAAI 2024 · similarity 0.86)
- Unsupervised Training Sequence Design: Efficient and Generalizable Agent Training (AAAI 2024 · similarity 0.86)
- Procedural Memory Distillation: Online Reflection for Self-Improving Language Models (NeurIPS 2026 · similarity 0.86)
- Reinforcement Learning Teachers of Test Time Scaling (NeurIPS 2025 · similarity 0.86)
- RLP: Reinforcement as a Pretraining Objective (ICLR 2026 · similarity 0.85)
- Diffusion-based Curriculum Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.