Research map / Large language models / Retrieval and knowledge
TableBench: A Comprehensive and Complex Benchmark for Table Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TableBench, an 886-sample industrial-style TableQA benchmark across 18 subcategories in four categories, plus TableInstruct (~20K SFT samples) and TableLLM fine-tuned open models that approach GPT-3.5 performance.
Problem
Existing table QA benchmarks do not capture the complex multi-step reasoning required by real-world industrial tabular tasks, leaving a gap between academic benchmarks and practical applications.
Approach
A manually curated, complexity-focused TableQA benchmark spanning fact checking, numerical reasoning, data analysis, and visualization, built with a GPT-4 agent self-inspiration question generator and a self-consistency answer annotation pipeline across three reasoning methods (TCoT, SCoT, PoT), plus an SFT corpus TableInstruct used to train TableLLM.
Key results
- GPT-4-Turbo achieves the best overall TCoT score of 51.32 on TableBench, while human performance is 85.91.
- TableLLM fine-tuned on CodeQwen-7B, Deepseek-Coder-7B, Llama3.1-8B, and Qwen2-7B reaches overall TCoT scores of 32-38, comparable to GPT-3.5-Turbo (30.85 TCoT overall).
- Using only 60% of TableInstruct retains over 90% of full-data performance; Llama-3-8B with under 4,000 samples surpasses Qwen1.5-70B on the benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune open-source models on TableInstruct | TableLLM · TableBench Overall | ROUGE-L | Helped |
| Use TCoT instead of direct prompting on small-size LLMs | Small-size LLMs · TableBench | overall score | Hurt |
| Use PoT instead of direct prompting on small-size LLMs | Small-size LLMs · TableBench | overall score | Hurt |
| Use PoT for numerical computation rather than textual reasoning | LLMs · TableBench Numerical Reasoning | ROUGE-L | Mixed |
| Execute PoT-generated code only once, without code correction | PoT · TableBench | executable code ratio: 78.67% | Hurt |
| Train with 60% of TableInstruct samples instead of the complete dataset | TableLLM · TableBench | relative performance: 60%; over 90% | Hurt |
Surprising findings
- Direct prompting (non-reasoning) outperforms reasoning-based methods for small-size LLMs, which struggle with PoT and SCoT.
- For GPT-4-Turbo, only 78.67% of PoT outputs were executable code, limiting PoT performance.
Limitations
- Focuses on reasoning complexity of questions rather than structural complexity of tables.
- Image-format tabular data is not covered.
- Answer scoring relies mainly on ROUGE-L, which may not capture flexible LLM outputs.
Experimental setup
Datasets: WTQ SQA TabFact FeTaQA FinQA AIT-QA WikiSQL Spider Bird Text2Analysis
Benchmarks: TableBench
Metrics: ROUGE-L pass@1 parsing ratio
Baselines: GPT-4-Turbo GPT-4o GPT-3.5-Turbo Qwen-Max Yi-Large GLM-4 Deepseek-Chat-V2 Deepseek-Coder-V2 StructLM Llama3.1-70B
Builds on: chain-of-thought program-of-thoughts in-context learning
Closest papers by meaning
- Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries (AAAI 2024 · similarity 0.92)
- Accurate and Regret-Aware Numerical Problem Solver for Tabular Question Answering (AAAI 2025 · similarity 0.92)
- Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding (ICLR 2024 · similarity 0.90)
- NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables (NeurIPS 2025 · similarity 0.89)
- TableRAG: Million-Token Table Understanding with Language Models (NeurIPS 2024 · similarity 0.89)
- CABINET: Content Relevance-based Noise Reduction for Table Question Answering (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.