Research map / Large language models / Agents, code and math

Can LLMs Reliably Simulate Human Learner Actions? A Simulation Authoring Framework for Open-Ended Learning Environments

AAAI 2025 · Large language models › Agents, code and math · approach: LLM planning with symbolic tools · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Hyp-Mix, a framework that authors LLM-based learner simulations by combining testable marginal distributional hypotheses, and tests GPT-4 Turbo's calibration retention across learner-model edits in the HoloOrbits physics environment.

Problem

LLM-simulated learner actions in open-ended learning environments are sensitive to prompt wording, may regurgitate memorized content, and can be steered by experts (Clever Hans effect), so their reliability for simulation authoring is unestablished.

Approach

Express expert knowledge as marginal distributional hypotheses grouped into hypothesis classes, each with a prompt template calibrated via statistical tests; then check whether calibration holds across five learner-model edit operations without re-engineering the prompt.

Key results

• GPT-4 Turbo maintained calibration in 16 of 18 cases across the learner model edit operations (Ex-Situ Transfer, Variable Swap, LC Swap, Calibration Regression), failing only on Combine Hypotheses for two of three action spaces.
• Learners with minimal geometry proficiency (1/10) produced ~80% productive measurements before a uniform-distribution hypothesis was added, illustrating hyper-accuracy distortion.

Measured effects reported in the paper

Surprising findings

• Minimal-proficiency learners initially produced ~80% productive measurements, contrary to expected uniform behavior, suggesting hyper-accuracy distortion.
• Combining separately calibrated hypotheses was the only operation where calibration was not maintained for two of three action spaces.

Limitations

• Only two learner characteristics (geometry proficiency and persistence) in a single environment (HoloOrbits).
• Only GPT-4 Turbo was tested; generalization to other LLMs, including open-source ones, is untested.
• Combining separately calibrated hypotheses failed calibration in two of three action spaces.

Experimental setup

Benchmarks: HoloOrbits

Metrics: Spearman correlation p-value Chi-squared test p-value

Builds on: chain-of-thought prompting Skill-Mix

Closest papers by meaning

• Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators (NeurIPS 2026 · similarity 0.90)
• OdysSim: Building Foundation Models for Human Behavior Simulation (NeurIPS 2026 · similarity 0.88)
• BEAGLE: Behavior-Enforced Agent for Grounded Learner Emulation (NeurIPS 2026 · similarity 0.88)
• Position: LLM Social Simulations Are a Promising Research Method (ICML 2025 · similarity 0.88)
• CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.87)
• Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces (NeurIPS 2026 · similarity 0.87)