Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Latent Learning Progress Drives Autonomous Goal Selection in Human Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Human participants (N=175) chose their own goals in a hierarchical alchemy task, and computational modeling showed goal selection is driven by a latent learning progress signal (distance to solution) beyond observed performance and standard learning progress.
Problem
What signals guide human autonomous goal selection, in particular whether humans use learning progress that is inferred internally without observable performance changes.
Approach
Proposes latent learning progress (LLP), measured as the fraction of the remaining action-sequence space still untested for the current goal, which updates with every new attempt even without positive feedback, and shows it predicts human goal choices better than standard learning progress (LP) in a hierarchical goal-selection task.
Key results
- The performance + LLP model was the most frequently expressed model (PXP = 1; average responsibility R = 0.63), versus 0.12 for performance + LP.
- Hierarchical complex goals (G3) were chosen less than non-hierarchical complex goals (G4) (M = 0.18 vs 0.21, p = 0.002), and learning was faster for hierarchical goals.
- Four behavioral clusters emerged; LLP was a central goal-selection component in all clusters, with cluster 2 (N=75, efficient learners) weighting LLP more strongly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add latent learning progress to performance-based goal values | Human goal selection model · Goal selection | average responsibility: 0.12 -> 0.63 | Helped |
| Use latent learning progress rather than standard learning progress alongside performance | Human goal selection model · Goal selection | average responsibility: 0.12 -> 0.63 | Helped |
| Use hierarchical goals rather than matched non-hierarchical goals | Human learners · Learning phase, complex goals | performance: 0.29 ± 0.02 -> 0.37 ± 0.02 | Helped |
| Use hierarchical goals rather than matched non-hierarchical goals | Human learners · Learning phase, compound goals | performance: 0.49 ± 0.02 -> 0.53 ± 0.02 | Helped |
| Use hierarchical rather than non-hierarchical complex goals | Human learners · Testing phase, complex goals | performance: 0.52 ± 0.03 -> 0.56 ± 0.03 | No clear effect |
| Use hierarchical rather than non-hierarchical compound goals | Human learners · Testing phase, compound goals | performance: 0.73 ± 0.03 -> 0.73 ± 0.03 | No clear effect |
| Choose 4-action rather than 2-action goals | Human participants · Goal selection | selection probability: 0.15 ± 0.00 -> 0.19 ± 0.01 | Helped |
| Use the 3-successful-attempt cutoff instead of the 1-successful-attempt cutoff for learning a goal (ablation) | Performance + LLP model · Goal selection model comparison | winning model XP: 1 -> 1 | No clear effect |
| Use a Bayesian threshold instead of a 1-successful-attempt cutoff to define learning (ablation) | Performance + LLP model · Goal selection model comparison | winning model XP: 1 -> 1 | No clear effect |
| Represent LLP as the proportion of possible solutions tested rather than the remaining solution space (ablation) | Performance + LLP model · Goal selection model comparison | winning model XP: 1 -> 1 | No clear effect |
| Add a perseveration term to the goal valuation models (ablation) | Goal selection models · Goal selection model comparison | winning model XP: 1 -> 1 | No clear effect |
| Use an alternative hierarchy coding scheme for the hierarchy factor (ablation) | Goal selection models · Goal selection model comparison | winning model XP: 1 -> 0.96 | No clear effect |
Surprising findings
- Hierarchy affected goal selection mainly indirectly through learning and LLP, not as a direct factor, since the winning model lacked direct hierarchy information.
- The standard learning progress model explained participant choices poorly (R = 0.12), below the performance-plus-LLP model.
Limitations
- Participants chose from a predefined menu of six goals rather than inventing their own goals.
- Goal selection was modeled separately from action selection; action choices were taken from data rather than jointly modeled.
- Sample was a relatively homogeneous group of undergraduate students (76% female, mostly ages 18-25).
Experimental setup
Datasets: human behavioral data collected in the alchemist potion-making task (N=175)
Benchmarks: human goal selection choices in the alchemist task
Metrics: protected exceedance probability average model responsibility Spearman correlation parameter recovery model recovery
Baselines: performance-only model performance + learning progress model performance + hierarchy model performance + LLP + hierarchy model
Builds on: learning progress intrinsic motivation options framework delta rule value learning
Closest papers by meaning
- Language Model Goal Selection Differs from Humans' in a Self-Directed Learning Task (NeurIPS 2026 · similarity 0.88)
- Identifying Latent State-Transition Processes for Individualized Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- LAGMA: LAtent Goal-guided Multi-Agent Reinforcement Learning (ICML 2024 · similarity 0.85)
- Beyond Single Stationary Policies: Meta-Task Players as Naturally Superior Collaborators (NeurIPS 2024 · similarity 0.85)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.