Research map / Large language models / Agents, code and math
Can large language models explore in-context?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Tests whether off-the-shelf LLMs (GPT-3.5, GPT-4, Llama2-13B) explore in-context in multi-armed bandits across 32 prompt designs, finding only GPT-4 with summarized history and reinforced chain-of-thought explores robustly.
Problem
Whether contemporary LLMs, without training interventions, can perform exploration in in-context multi-armed bandit decision making, where history and environment are specified only in the prompt.
Approach
Deploys LLMs as bandit agents via a factorial space of prompt designs and uses surrogate statistics (suffix failure frequency and scaled minimum arm fraction) that can be measured at moderate scale to diagnose long-term exploration failure, rather than noisy cumulative reward.
Key results
- Only GPT-4 with buttons scenario, suggestive framing, summarized history, reinforced CoT, temperature 0 (BSS C0) avoids both suffix and uniform-like failures; its behavior resembles Thompson Sampling.
- GPT-4 with basic prompt (BNRN0) shows suffix failures in more than 60% of replicates over T=500 rounds.
- GPT-3.5 and Llama2 perform worse than GPT-4 across configurations; GPT-4-BSR C0 (raw history ablation) exhibits non-trivial suffix failures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the BSS reinforced-CoT prompt: suggestive framing, externally summarized history, and chain-of-thought at temperature 0 | GPT-4 · Hard 5-armed bandit | exploration success | Helped |
| Replace summarized history with raw history (ablation) | GPT-4 BSS reinforced-CoT configuration · Hard 5-armed bandit, T=200 | suffix failure frequency | Hurt |
| Use the basic prompt instead of the BSS reinforced-CoT prompt | GPT-4 · Hard 5-armed bandit, T=500 | suffix failure frequency: >60% | Hurt |
| Use GPT-3.5 or Llama2 instead of GPT-4 | LLM agents · Hard multi-armed bandit | exploration performance | Mixed |
Surprising findings
- Removing external summarization of history from the otherwise successful GPT-4 configuration causes exploration failure.
- Per-round decision statistics are highly sensitive to the data source, making it hard to diagnose whether LLM agents are too greedy or too uniform.
Limitations
- Only one successful configuration found, which relies on externally summarized history that may not scale to complex or contextual settings.
- Results are sensitive to small prompt changes and statistical power is limited (N in {10,20}, T=100 for most experiments).
- Only simple bandit instances (K=5, gap 0.2) were studied, and only three LLMs, some of which are closed API models.
Experimental setup
Benchmarks: stochastic Bernoulli bandit (K=5, gap 0.2, hard) stochastic Bernoulli bandit (K=4, gap 0.5, easy)
Metrics: suffix failure frequency MinFrac (uniform-like failure) median rescaled reward GreedyFrac
Baselines: UCB Thompson Sampling Greedy epsilon-Greedy
Builds on: UCB Thompson Sampling zero-shot chain-of-thought
Closest papers by meaning
- EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration (ICML 2025 · similarity 0.93)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.92)
- Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.91)
- From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.91)
- On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.91)
- Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.