Research map / Large language models / Language, safety and interpretability
In-Context Learning Agents Are Asymmetric Belief Updaters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLMs (Claude v1.3) and meta-RL-trained transformer agents update beliefs asymmetrically in two-alternative forced choice tasks: optimism bias for chosen options, reversed for counterfactual unchosen feedback, and absent without agency.
Problem
How in-context learning in LLMs updates beliefs from outcomes, and whether the asymmetric (optimism-type) belief updating seen in humans also appears in LLMs and idealized in-context learners, depending on problem framing and agency.
Approach
Applies cognitive-science analysis (Rescorla-Wagner with separate positive/negative learning rates, fitted via MAP and compared with BIC/posterior probability) to the choice behavior of an LLM in 2AFC tasks, and replicates the patterns in meta-RL-trained transformer agents to test whether asymmetric updating is rational.
Key results
- In partial feedback, LLM learning rates were alpha+ = 0.87 and alpha- = 0.10 (RW± preferred, PP 0.93), an optimism bias.
- In full feedback, the chosen option showed optimism while the unchosen option showed pessimism; a 2-learning-rate confirmation model fit better (PP 0.92).
- Without agency (forced-choice trials), updates were symmetric (3α model PP 0.77), while free-choice updates were asymmetric (alpha+ = 0.35 vs alpha- = 0.08).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Update from positive versus negative prediction errors in the default partial-feedback task | Claude v1.3 · 2AFC partial feedback | prediction-error learning rate: $\alpha^+ = 0.87 \pm 0.01, \alpha^- = 0.10 \pm 0.04$ | Mixed |
| Reveal outcomes for the unchosen option | Claude v1.3 · 2AFC full feedback | final regret: $0.07 \pm 0.01$ -> $0.02 \pm 0.01$ | Helped |
| Use the confirmatory 2α model instead of the four-rate full-feedback model (ablation) | Claude v1.3 belief-update analysis · 2AFC full feedback | posterior probability: $0.92 \pm 0.03$ | Helped |
| Add forced-choice observations made by someone else | Claude v1.3 · 2AFC agency condition | regret: $0.07 \pm 0.2$ to $0.02 \pm 0.01$ | Helped |
| Separate positive and negative learning rates for forced-choice trials (ablation) | Claude v1.3 4α versus 3α model · 2AFC agency condition | model posterior probability: $\text{PP}_{3\alpha} = 0.77 \pm 0.06$ | Hurt |
| Learn from positive versus negative prediction errors in free-choice trials, while updating symmetrically in forced-choice trials | Claude v1.3 · 2AFC agency condition | prediction-error learning rate: $\alpha^{+} = 0.35 \pm 0.06$; $\alpha^{-} = 0.08 \pm 0.01$ | Mixed |
| Train a transformer agent with Meta-RL rather than use the LLM | 2AFC task suite · 2AFC tasks | regret | Helped |
Surprising findings
- The LLM's asymmetric updating reverses for counterfactual (unchosen) feedback, showing a pessimism-like pattern for unchosen options.
- The asymmetry disappears when the outcome is observed for another agent's forced choice rather than the model's own choice.
Limitations
- Single LLM (Claude v1.3) evaluated, limiting generality.
- Causal link between fitted learning rates and the underlying mechanism is unclear; alternative auxiliary processes possible.
- Highly controlled synthetic bandit settings limit external validity.
Experimental setup
Datasets: human data from prior studies (reproduced) simulated LLM behavior on 2AFC tasks simulated meta-RL agent behavior
Benchmarks: 2AFC partial feedback task 2AFC full feedback task 2AFC agency (mixed-choice) task
Metrics: regret learning rates (alpha+, alpha-) posterior probability from BIC model comparison
Baselines: Rescorla-Wagner model human participants
Builds on: Rescorla-Wagner model RW± model meta-reinforcement learning actor-critic
Closest papers by meaning
- Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence (ICML 2025 · similarity 0.88)
- Reflection-Bench: Evaluating Epistemic Agency in Large Language Models (ICML 2025 · similarity 0.88)
- Sparse Autoencoders Reveal Temporal Difference Learning in Large Language Models (ICLR 2025 · similarity 0.88)
- The Lock-in Hypothesis: Stagnation by Algorithm (ICML 2025 · similarity 0.87)
- Can large language models explore in-context? (NeurIPS 2024 · similarity 0.87)
- Large Language Model Agents Are Not Always Faithful Self-Evolvers (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.