Research map / Large language models / Agents, code and math
MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MINT, a benchmark of 586 repurposed tasks that measures LLMs' multi-turn task solving with Python tool use and GPT-4-simulated natural language feedback across 20 open and closed LLMs.
Problem
Current LLM evaluations are mostly single-turn and ignore tool use and natural language feedback from users, so they miss how models perform in realistic multi-turn user-LLM-tool interaction.
Approach
Build a compact multi-turn evaluation harness where the model solves tasks by executing Python code and receives feedback simulated by GPT-4 (validated against human feedback), and measure success-rate gains per interaction turn and per feedback turn.
Key results
- All models benefit from tool interaction and language feedback, with gains of 1-8% per extra tool turn and 2-17% with feedback.
- gpt-4-0613 achieves 69.5% SR at k=5 without feedback, the best overall; best open model Lemur-v1 70B SIFT reaches 43.7% with feedback, 8.7% behind the best closed model claude-instant-1 (52.4%).
- GPT-4 simulated feedback was rated more helpful than human feedback in 54.9% of comparisons, versus 8.8% for human feedback.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| allow additional tool-interaction turns | LLMs evaluated on MINT · MINT tasks, k=1 to k=5 | success rate: 1--8% | Helped |
| add GPT-4 natural-language feedback | LLMs evaluated on MINT · MINT tasks, k=5 | success rate: 2--17% | Helped |
| replace base training with SIFT | CodeLLaMA-34B · MINT multi-turn, with and without feedback | success rate: 11.1%; 15.4% | Hurt |
| replace base training with RLHF | LLaMA-2-70B · MINT multi-turn, with and without feedback | success rate: 8.5%; 8.7% | Hurt |
| fine-tune on multi-turn conversation data | Vicuna-v1.5 7B and Lemur-v1 70B · MINT multi-turn task solving | success rate: Vicuna-v1.5 (7B) ... $12.6$% vs. ... $9.7$% / $7.3$%; Lemur-v1 ... stronger performance than its Base variant. | Helped |
| add ground-truth information to feedback generation (ablation) | GPT-4 feedback for GPT-3.5-turbo-0613 · MINT reasoning and code generation | feedback gain: +3.43; +8.82 | Helped |
| add ground-truth information to feedback generation (ablation) | GPT-4 feedback for GPT-3.5-turbo-0613 · MINT decision-making | feedback gain: -33.58 | Hurt |
| replace textual feedback with binary feedback (ablation) | GPT-3.5-turbo-0613 · MINT micro average | feedback gain: -12.38 | Hurt |
| provide binary feedback instead of no feedback (ablation) | GPT-3.5-turbo-0613 · MINT decision-making | feedback gain: +5.97 | Helped |
| change dense feedback to sparse feedback (ablation) | GPT-3.5-turbo-0613 · MINT micro average | feedback gain: -11.02 | Hurt |
| provide sparse feedback instead of no feedback (ablation) | GPT-3.5-turbo-0613 · MINT tasks | success rate: +4.07 | Helped |
| use GPT-4-generated feedback instead of human-written feedback | gpt-3.5-turbo-0613 · MINT tasks, k=5 | success rate: 32.7% vs. 33.6% | No clear effect |
Surprising findings
- SIFT and RLHF generally hurt multi-turn performance, e.g., RLHF lowers LLaMA-2 70B success by 8.5% and 8.7%.
- Better single-turn performance does not guarantee better multi-turn performance; claude-instant-1 overtakes claude-2 with more turns.
Limitations
- GPT-4 simulated feedback may not cover the full range of real human responses, especially for nuanced value judgments.
- Metrics focus on outcomes and may not capture interaction quality, such as models guessing repeatedly.
- Tool coverage is limited to Python execution and a Wikipedia search tool, excluding web browsing and OS tasks.
Experimental setup
Datasets: HumanEval MBPP ALFWorld GSM8K HotpotQA MATH MMLU TheoremQA ShareGPT
Benchmarks: MINT HumanEval MBPP ALFWorld GSM8K HotpotQA MATH MMLU TheoremQA
Metrics: success rate improvement per turn (regression slope) delta feedback
Baselines: gpt-4-0613 gpt-3.5-turbo-0613 claude-2 claude-instant-1 chat-bison-001 LLaMA-2 CodeLLaMA Vicuna-v1.5 Lemur-v1
Builds on: ReAct Reflexion ALFWorld HumanEval MBPP
Closest papers by meaning
- EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solving (NeurIPS 2025 · similarity 0.90)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.90)
- Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap (NeurIPS 2026 · similarity 0.90)
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.90)
- MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use (ICLR 2024 · similarity 0.90)
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.