Research map / Multi-agent systems and games / LLM agents
Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces COTA, a multi-turn conversational tabular data analysis benchmark built with a GPT-4 multi-agent sandbox (DECISION COMPANY) plus expert annotation, and proposes Adaptive Conversation Reflection (ACR) to improve LLM agents.
Problem
Realistic conversational logs for tabular data analysis are costly and privacy-sensitive to collect, so LLMs lack comprehensive quantitative evaluation for multi-turn data analysis with clarification, private libraries, and actions.
Approach
Generate conversational data analysis logs with a multi-agent sandbox (GPT-4 agents under human expert supervision) and evaluate LLMs across four modes (Normal, Action, Private, Private Action), plus a self-generated pseudo-code reflection method (ACR) that makes agents learn from their own successful histories.
Key results
- Claude-3.5-Sonnet with Inter-Agent (ACR) reaches 40.0% overall on COTA, the best reported, versus 29.6% base.
- ACR gives Claude-3.5-Sonnet a relative improvement of up to 35.14% over its base model.
- Human expert evaluation after human involvement shows about 0.95 acceptance ratio on conversation quality metrics, versus about 0.19-0.67 without human involvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add tool use and COT/CodeAct reasoning | Claude-3.5-Sonnet · COTA Overall | accuracy: 29.6 -> 32.1 | Helped |
| Add textual reflection to Agent | Claude-3.5-Sonnet Agent · COTA Overall | accuracy: 32.1 -> 34.9 | Helped |
| Use multiple specialized collaborating agents | Claude-3.5-Sonnet Agent · COTA Overall | accuracy: 32.1 -> 36.4 | Helped |
| Add Adaptive Conversation Reflection (ACR) | Claude-3.5-Sonnet Multi-Agent · COTA Overall | accuracy: 36.4 -> 40.0 | Helped |
| Add Adaptive Conversation Reflection (ACR) | Claude-3.5-Sonnet · COTA Action | accuracy: 36.3 -> 52.8 | Helped |
| Add Adaptive Conversation Reflection (ACR) | Claude-3.5-Sonnet · COTA Private | accuracy: 10.5 -> 18.0 | Helped |
| Add Adaptive Conversation Reflection (ACR) | Claude-3.5-Sonnet · COTA Pri-Act | accuracy: 9.3 -> 14.8 | Helped |
| Add Adaptive Conversation Reflection (ACR) | Claude-3.5-Sonnet · COTA Normal | accuracy: 37.9 -> 42.9 | Helped |
| Use Inter-Agent with ACR instead of Agent | Claude-3.5-Sonnet · COTA Best Guess action | accuracy | Hurt |
| Add human expert annotation to the sandbox-generated dataset | DECISION COMPANY annotations · COTA dataset quality | acceptance ratio: 0.19 -> 0.97 | Helped |
| Add human expert annotation to the sandbox-generated dataset | DECISION COMPANY annotations · COTA dataset quality | acceptance ratio: 0.46 -> 0.96 | Helped |
Surprising findings
- Claude-3.5-Sonnet and Llama-3.3-70B outperform GPT-4 models in base mode, suggesting the benchmark is not overfit to the GPT family.
- ACR makes agents overly reliant on historical data, which hurts performance on Best_Guess actions where agents select 'None of Above'.
Limitations
- Top score of 40.0% indicates the benchmark is far from solved, but results rely on a single benchmark built from five domains.
- Data is generated by GPT-4 agents, so human supervision is needed to reach quality; without it acceptance is low.
- ACR can make agents overly tractable and hurt best-guess behavior.
Experimental setup
Datasets: Kaggle tabular datasets (ATP Tennis, Credit Card, Fast Food, Laptop Price, Melbourne Housing)
Benchmarks: COTA
Metrics: Accuracy Accuracy with Private Library Recall (AccR)
Baselines: Mistral-7B Mistral-8x7B CodeLlama-34B GPT-4-Turbo Claude-3-Opus GPT-4-32k Llama-3.3-70B Claude-3.5-Sonnet
Builds on: CodeAct Chain-of-Thought prompting BIRD ARCADE
Closest papers by meaning
- TableBench: A Comprehensive and Complex Benchmark for Table Question Answering (AAAI 2025 · similarity 0.87)
- Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries (AAAI 2024 · similarity 0.86)
- NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables (NeurIPS 2025 · similarity 0.86)
- Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models (NeurIPS 2024 · similarity 0.86)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.86)
- LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.