Research map / Multi-agent systems and games / Human-AI and goal-conditioned

Can In-Context Learning Support Intrinsic Curiosity?

NeurIPS 2026 · Multi-agent systems and games › Human-AI and goal-conditioned · approach: Intrinsically motivated exploration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that in-context learners' prediction errors can yield curiosity rewards that approximate Bayesian information gain, proving impossibility for general BAMDPs and asymptotic recovery in Bayesian Experimental Design settings, validated on GP, Mastermind, and Alchemy.

Problem

Automated data selection (intrinsic curiosity) needs learning-progress rewards, which normally require expensive inner-loop gradient updates of a world model; the question is whether in-context learning of sequence models can provide cheap, update-free predictions for such rewards that approximate B

Approach

Uses a pretrained in-context learner (PFN) as an amortized Bayesian posterior predictive, and defines prediction-error-based curiosity rewards (surprisal, description-length learning progress, and a new telescoping masked-context reward R_sum) computable from its predictions. Proves R_sum is a lower bound on BIG that asymptotically converges in BED settings, and that no finite-horizon reward in th

Key results

• In the noisy GP environment, R_sum and R_dl achieve validation performance equivalent to training on the task reward, while surprisal performs significantly worse than a random policy.
• In noisy Mastermind and Alchemy, surprisal-trained policies fall below the random baseline, whereas R_sum and R_dl remain robust.
• In noiseless Alchemy under the oracle and PFN, R_sum achieves the highest validation log-probability in the noisy variant, maintaining superiority over other rewards.

Surprising findings

• Surprisal-based curiosity can fall below a random policy in noisy environments, because it seeks aleatoric noise (noisy TV problem).
• Prediction-based learning progress rewards are provably non-implementable with an in-context learner's prediction errors for general BAMDPs.

Limitations

• General BAMDP case lacks a practical unbiased ICL-implementable reward; theory relies on the in-context learner being exactly Bayesian.
• Experiments are limited to episodic BED environments with pretrained frozen PFNs; policy-induced covariate shift is not fully addressed.
• Rewards are computed only at episode end, limiting online RL applicability.

Experimental setup

Datasets: synthetic trajectories from Gaussian Process priors synthetic Mastermind trajectories Alchemy environment trajectories

Benchmarks: Gaussian Process function estimation Mastermind Alchemy

Metrics: validation log-likelihood validation probability of true code validation predictive log-probability BIG across trajectories

Baselines: surprisal description-length learning progress task reward (validation metric) random policy

Builds on: NDIGO Bayesian information gain (Bayesian surprise) learning progress prior-fitted networks TabPFN Alchemy

Closest papers by meaning

• Competition Dynamics Shape Algorithmic Phases of In-Context Learning (ICLR 2025 · similarity 0.88)
• Can large language models explore in-context? (NeurIPS 2024 · similarity 0.87)
• In-Context Learning Strategies Emerge Rationally (NeurIPS 2025 · similarity 0.87)
• In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought (ICML 2024 · similarity 0.87)
• Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context (NeurIPS 2025 · similarity 0.87)
• In-context Exploration-Exploitation for Reinforcement Learning (ICLR 2024 · similarity 0.87)