Research map / Large language models / Retrieval and knowledge
Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Text2Analysis, a benchmark of 2249 (table, query, code, result) tuples over 347 tables covering advanced analysis (insights, forecasting, chart generation) and unclear queries, built with five LLM-assisted annotation methods and evaluated on GPT-4, StarChat, CodeGen2.5 and TAPEX.
Problem
Existing table QA and Text2SQL benchmarks focus on rudimentary descriptive operations and clear queries, leaving advanced analysis (insights, forecasting, chart generation) and unclear real-world user queries under-evaluated.
Approach
Extends table-to-code benchmarks beyond SQL-compatible operations to advanced analysis tasks and deliberately removes query parameters to create unclear queries, while using five LLM-assisted annotation pipelines (forward annotation, reverse generation from code or results, table expansion, and unclear-query expansion) to scale annotation.
Key results
- GPT-4 achieves the highest overall executable code ratio (69.46%) and pass@1 (41.01%) among evaluated models.
- On forecasting, GPT-4 and StarChat-beta reach pass@1 of 35.71% and 2.33% respectively, with ECR below 16% for all models.
- Human evaluation of 100 samples yields an overall average score of 0.91 and Cohen's Kappa of 0.79.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 instead of the evaluated code-generation models | Text2Analysis · Text2Analysis overall | pass@1: GPT-4: 41.01%; code-generation models: 24.86% lower | Helped |
| Use GPT-4 instead of other evaluated models | Text2Analysis · Text2Analysis overall | pass@1: 41.01% | Helped |
| Omit the measure field from the query | GPT-4 · Basic Insights unclear queries | ECR: decreases by 8% | Hurt |
| Omit the measure field from the query | GPT-4 · Chart Generation unclear queries | pass rate: decreased by 25% | Hurt |
| Use a clear query rather than an unclear query | GPT-4 · Chart Generation | pass rate: 86% | Helped |
| Generate forecasting code with complex library operations | GPT-4 · Forecasting | ECR: below 1% | Hurt |
| Generate forecasting code with complex library operations | GPT-4 · Forecasting | pass rate: below 1% | Hurt |
| Use GPT-4 rather than StarChat-β for forecasting | Text2Analysis · Forecasting | CORR: 0.10 vs 0.16 | Mixed |
Surprising findings
- Code generation models match GPT-4 on executable code ratio but have much lower pass rates.
- Forecasting ECR and pass rate are below 1% for most models in the unclear-query setting, with GPT-4 generating incorrect parameters in over 50% of forecasting cases.
Limitations
- Focuses on proposing a benchmark and does not explore solution methods for unclear queries in depth.
- Ambiguous-parameter and unclear-task query types are excluded from the evaluation.
- Only five baseline models evaluated, with limited coverage of recent open LLMs.
Experimental setup
Datasets: Text2Analysis (constructed) AnaMeta chart corpus crawled data analysis library code snippets
Benchmarks: Text2Analysis
Metrics: executable code ratio pass@1 CORR RMSE MAE MedAE
Baselines: GPT-4 StarChat-alpha StarChat-beta CodeGen2.5 TAPEX
Builds on: Text2SQL TableQA TAPEX
Closest papers by meaning
- TableBench: A Comprehensive and Complex Benchmark for Table Question Answering (AAAI 2025 · similarity 0.92)
- Accurate and Regret-Aware Numerical Problem Solver for Tabular Question Answering (AAAI 2025 · similarity 0.88)
- TableRAG: Million-Token Table Understanding with Language Models (NeurIPS 2024 · similarity 0.87)
- MMTU: A Massive Multi-Task Table Understanding and Reasoning Benchmark (NeurIPS 2025 · similarity 0.87)
- RADAR: Benchmarking Language Models on Imperfect Tabular Data (NeurIPS 2025 · similarity 0.87)
- Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.