Research map / Multi-agent systems and games / LLM agents
InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces InsightBench, a 100-dataset benchmark with planted insights for evaluating LLM agents on end-to-end multi-step business data analytics, scored with an open-source LLaMA-3-70b evaluator, and proposes the AgentPoirot baseline agent.
Problem
Existing data science benchmarks evaluate single-step code completion or single queries, and end-to-end analytics agents have only been assessed with costly, inconsistent human evaluation. A standardized automated benchmark for multi-step insight generation is needed.
Approach
Plant ground-truth insights (trends) into synthetic ServiceNow-style enterprise tables with expert-annotated analysis notebooks, and score agent-generated insights and summaries with a LLaMA-3-70b G-Eval variant in a one-to-many matching scheme, alongside an agent (AgentPoirot) that generates descriptive, diagnostic, predictive and prescriptive follow-up questions.
Key results
- AgentPoirot with gpt-4o reaches an insight-level LLaMA-3-Eval score of 0.60 versus 0.54 for Pandas Agent with gpt-4o.
- Using a generic goal instead of the SMART goal drops AgentPoirot (gpt-4o) insight-level LLaMA-3-Eval from 0.60 to 0.40.
- AgentPoirot with LLaMA-3-70b (0.52 insight-level) is close to gpt-4-turbo (0.56) and above gpt-3.5-turbo (0.50).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AgentPoirot instead of Pandas Agent (gpt-4o) | Analytics agent · InsightBench | Insight-level LLaMA-3-Eval: 0.54 -> 0.60 | Helped |
| Use AgentPoirot instead of Pandas Agent (gpt-4o) | Analytics agent · InsightBench | Insight-level ROUGE-1: 0.35 -> 0.32 | Hurt |
| Use AgentPoirot instead of Pandas Agent (gpt-4o) | Analytics agent · InsightBench | Summary-level LLaMA-3-Eval: 0.40 -> 0.44 | Helped |
| Use AgentPoirot instead of Pandas Agent (gpt-4o) | Analytics agent · InsightBench | Summary-level ROUGE-1: 0.35 -> 0.37 | Helped |
| Use a carefully designed goal instead of a generic goal (ablation) | AgentPoirot · InsightBench | Insight-level LLaMA-3-Eval: 0.40 -> 0.60 | Helped |
| Use a carefully designed goal instead of a generic goal (ablation) | AgentPoirot · InsightBench | Summary-level LLaMA-3-Eval: 0.33 -> 0.44 | Helped |
| Generate diverse follow-up questions instead of only one type (ablation) | AgentPoirot (llama-3-70b) · InsightBench | Insight-level ROUGE-1: 0.29 -> 0.33 | Helped |
| Generate diverse follow-up questions instead of only one type (ablation) | AgentPoirot (llama-3-70b) · InsightBench | Insight-level LLaMA-3-Eval: 0.51 -> 0.52 | Helped |
| Use many-to-many instead of one-to-many matching (ablation) | LLaMA-3-Eval · InsightBench first 10 datasets | Soft-score LLaMA-3-Eval: 0.57 -> 0.43 | Hurt |
| Use many-to-many instead of one-to-many matching (ablation) | LLaMA-3-Eval · InsightBench first 10 datasets | Soft-score LLaMA-3-Eval: 0.55 -> 0.40 | Hurt |
| Use many-to-many instead of one-to-many matching (ablation) | LLaMA-3-Eval · InsightBench first 10 datasets | Soft-score LLaMA-3-Eval: 0.50 -> 0.35 | Hurt |
| Use many-to-many instead of one-to-many matching (ablation) | LLaMA-3-Eval · InsightBench first 10 datasets | Soft-score LLaMA-3-Eval: 0.50 -> 0.34 | Hurt |
Surprising findings
- Trend detection showed no false positives: all models failed to flag insights when the planted slope was below 0.1, including negative slopes.
- Many-to-many LLM matching of ground-truth to predictions gave lower scores than one-to-many matching because it often mismatched insights.
Limitations
- Synthetic data may not capture genuine enterprise data complexity.
- Limited to five business themes across 100 datasets.
- Several baseline agents excluded due to unavailable code or closed access.
Experimental setup
Datasets: InsightBench (synthetic ServiceNow-style tables)
Benchmarks: InsightBench
Metrics: LLaMA-3-Eval score ROUGE-1 G-Eval score
Baselines: Pandas Agent ROUGE-1 G-Eval
Builds on: G-Eval Pandas Agent InsightPilot
Closest papers by meaning
- EnterpriseBench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance (NeurIPS 2026 · similarity 0.86)
- InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks (ICML 2024 · similarity 0.86)
- BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows (NeurIPS 2026 · similarity 0.86)
- DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle (ICLR 2026 · similarity 0.85)
- DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? (ICLR 2025 · similarity 0.84)
- DRBench: A Realistic Benchmark for Enterprise Deep Research (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.