Research map / Large language models / Instruction and fine-tuning

EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration

ICML 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces BanditBench, a natural-language multi-armed and contextual bandit benchmark for LLM in-context exploration, and improves LLMs via algorithm-guided UCB support and distillation of UCB trajectories through few-shot demos and fine-tuning.

Problem

LLMs are under-studied in sequential decision-making requiring exploration under uncertainty; it is unclear whether they can explore in-context to solve multi-armed and contextual bandit tasks.

Approach

Provide LLMs with UCB-style exploitation values and exploration bonuses in context (algorithm-guided support), and distill optimal UCB/LinUCB trajectories into LLMs via in-context demonstrations or oracle behavior fine-tuning (OFT), framed as next-action likelihood maximization.

Key results

• Gemini-1.5 Pro with algorithm-guided support reaches 59.6% MAB overall win-rate and 64.3% CB win-rate, versus 45.5% and 7.1% with raw history, while UCB scores 90.6% (MAB) and LinUCB 96.4% (CB).
• Oracle behavior fine-tuning of Gemini-1.5 Flash with raw history reaches 65.6% MAB win-rate, and fine-tuned Flash surpasses Gemini-1.5 Pro as stated by the authors.
• On CB, fine-tuning Flash with AG trajectories raises win-rate from 28.6% (RH) to 89.3% (AG at inference is used in evaluation as well).

Measured effects reported in the paper

Surprising findings

• Few-shot demonstrations helped Gemini-1.5 Flash but hurt Gemini-1.5 Pro in both MAB and CB.
• Fine-tuning on the hard (Clothes, K=20) trajectories generalized better than on easy trajectories, while few-shot demonstrations showed the opposite trend.

Limitations

• A substantial optimality gap remains between LLMs and UCB/LinUCB.
• Fine-tuning may overfit to training task distribution, with weaker generalization for easy-trained models on some settings.
• Evaluation cost is high for large API models, limiting scale of benchmark use.

Experimental setup

Datasets: MovieLens-1M UCB-generated bandit trajectories LinUCB-generated contextual bandit trajectories

Benchmarks: BanditBench MAB (16 configurations) BanditBench CB (MovieLens-based, K=10 and K=30)

Metrics: overall win-rate (pairwise Student's t-test, p<0.05) cumulative regret MinFrac OptFrac fitted regret function alpha and beta

Baselines: Raw History prompting Summarized History prompting UCB LinUCB

Builds on: UCB LinUCB Algorithm Distillation Behavior Cloning Summarized history from prior LLM bandit work

Closest papers by meaning

• Can large language models explore in-context? (NeurIPS 2024 · similarity 0.93)
• Efficient Exploration for LLMs (ICML 2024 · similarity 0.90)
• From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems (ICML 2024 · similarity 0.90)
• Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.90)
• Efficient Reinforcement Learning with Large Language Model Priors (ICLR 2025 · similarity 0.90)
• LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models (NeurIPS 2025 · similarity 0.90)