Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Training a Generally Curious Agent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Paprika fine-tunes LLM agents on synthetic multi-turn interaction trajectories from 10 task groups using DPO/RPO plus a UCB-based curriculum over task groups, yielding zero-shot transfer to unseen decision-making tasks.
Problem
LLMs often fail at strategic information gathering and exploration in multi-turn, partially observable decision-making tasks, and generalizing such skills to unseen tasks without task-specific training is hard.
Approach
Generate diverse self-sampled interaction trajectories across many textual task groups at high temperature, label winners/losers by task success and turn count, fine-tune with multi-turn SFT+DPO (RPO), and select task groups for training via a UCB bandit using a normalized-variance (coefficient of variation) learning-potential measure.
Key results
- Paprika improves Llama-3.1-8B-Instruct's bandit best-arm success rate from 42.25% to 62.25% after training only on other task groups.
- Averaged over the 10 task groups, Paprika increases Llama-3.1-8B-Instruct's success rate by 47% relative to its original performance with about 22,500 trajectories.
- Curriculum learning outperforms uniform task sampling by 1.4% average and 3.3% pass@4 success rate after three rounds on twenty questions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on trajectories from other task groups, leaving out bandit tasks | Llama-3.1-8B-Instruct · Bandit best arm selection | success rate: 42.25% to 62.25% | Helped |
| Use curriculum sampling rather than uniform sampling | Paprika on Llama-3.1-8B-Instruct · Twenty questions | average success rate: 1.4% | Helped |
| Use curriculum sampling rather than uniform sampling | Paprika on Llama-3.1-8B-Instruct · Twenty questions | pass@4 success rate: 3.3% | Helped |
| Fine-tune with Paprika | Llama-3.1-8B-Instruct · MT-Bench | score: 7.88 -> 8.14 (0.03) | Helped |
| Fine-tune with Paprika | Llama-3.1-8B-Instruct · AlpacaEval | score: 33.6 -> 33.5 (0.3) | No clear effect |
| Fine-tune with Paprika | Llama-3.1-8B-Instruct · GPQA | score: 33.5 -> 32.8 (1.5) | No clear effect |
| Fine-tune with Paprika | Llama-3.1-8B-Instruct · Math (Hard) | score: 24.6 -> 25.3 (0.3) | Helped |
| Fine-tune with Paprika | Llama-3.1-8B-Instruct · MMLU-Pro | score: 46.7 -> 46.2 (0.1) | No clear effect |
| Fine-tune with Paprika | Llama-3.1-8B-Instruct · IFEval | score: 84.4 -> 85.4 (0.3) | Helped |
| Train on all task groups rather than leave one group out | Paprika on Llama-3.1-8B-Instruct · 10 task groups | success rate across task groups | Mixed |
| Train on all task groups except the evaluation group | Paprika on Llama-3.1-8B-Instruct · Unseen task groups | success rate | Mixed |
Surprising findings
- Leave-one-out models trained without the target group often match or exceed group-specific training, though transfer is absent or negative for Wordle.
Limitations
- Requires a sufficiently capable starting model to generate successful trajectories within budget.
- Uses offline preference tuning rather than online RL due to compute constraints.
- Task environments required substantial human effort to design; curriculum depends on quality of task-group clustering.
Experimental setup
Datasets: synthetic trajectories from 10 task groups (Twenty questions, Guess my city, Wordle, Cellular automata, Customer service LMRL-Gym (topics for evaluation) MT-Bench AlpacaEval GPQA MATH hard MMLU-Pro IFEval
Benchmarks: Paprika task groups (held-out test splits) MT-Bench AlpacaEval GPQA MATH (Hard) MMLU-Pro IFEval
Metrics: average success rate pass@4 success rate average number of turns
Baselines: Llama-3.1-8B-Instruct UCB-based bandit fine-tuning uniform task sampling Single Task Group training
Builds on: DPO RPO STaR UCB Min-p sampling
Closest papers by meaning
- Don't Just Fine-tune the Agent, Tune the Environment (ICLR 2026 · similarity 0.89)
- Learn-by-interact: A Data-Centric Framework For Self-Adaptive Agents in Realistic Environments (ICLR 2025 · similarity 0.87)
- Recursive Introspection: Teaching Language Model Agents How to Self-Improve (NeurIPS 2024 · similarity 0.87)
- Learning POMDP World Models from Observations with Language-Model Priors (NeurIPS 2026 · similarity 0.87)
- Representation-Based Exploration for Language Models: From Test-Time to Post-Training (ICLR 2026 · similarity 0.87)
- Behavioral Exploration: Learning to Explore via In-Context Adaptation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.