Research map / Large language models / Retrieval and knowledge
Accurate and Regret-Aware Numerical Problem Solver for Tabular Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TabLaP, a multi-LLM TableQA system where GPT-3.5-based NumSolver plans Python calculations, a fine-tuned Llama3-8B selects between branches, and an LLM+UCB bandit module outputs trustworthiness labels.
Problem
TableQA with numerical questions is hard because LLMs make calculation errors on numerical values in tables; users also need a sense of when model answers can be trusted.
Approach
Use the LLM as a planner that writes Python scripts executed by an interpreter rather than computing answers directly, combine it with a SOTA TableQA branch via a fine-tuned answer selector, and add a bandit-based trustworthiness evaluator for regret-aware use.
Key results
- TabLaP reaches 76.6% accuracy on WTQ (vs 72.5% for Mix-SC) and 44.1% on FTQ (vs 41.7% for GPT-4o), a +5.7 and +5.8 point gain.
- TwEvaluator TwAccuracy of 77.9% on WTQ and 53.7% on FTQ, reducing user regret ratio by 19.6% and 20.6% relative to always trusting answers.
- On TabFact_small, TabLaP achieves 90.4% accuracy vs 88.6% for GPT-4o mini.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TabLaP instead of the strongest baseline, Mix-SC | TableQA · WTQ | accuracy: +5.7 | Helped |
| Use TabLaP instead of the strongest baseline, GPT-4o | TableQA · FTQ | accuracy: +5.8 | Helped |
| Use TabLaP instead of the strongest baseline | TableQA · TabFact_small | accuracy: +2.0 | Helped |
| Use NumSolver's Python-planning numerical branch with TabLaP | TabLaP · WTQ numerical questions | accuracy: 72.8% vs. 68.5% | Helped |
| Use NumSolver's Python-planning numerical branch with TabLaP | TabLaP · FTQ numerical questions | accuracy: 42.3% vs. 39.6% | Helped |
| Use trustworthiness labels rather than always trusting model answers | TabLaP · WTQ | user regret ratio: 19.6% | Helped |
| Use trustworthiness labels rather than always trusting model answers | TabLaP · FTQ | user regret ratio: 20.6% | Helped |
| Use NumSolver alone instead of the dual-branch TabLaP (ablation) | TabLaP · WTQ | accuracy: 76.6 -> 64.7 | Hurt |
| Use NumSolver alone instead of the dual-branch TabLaP (ablation) | TabLaP · FTQ | accuracy: 44.1 -> 42.6 | Hurt |
| Add NumSolver instead of replacing it with GPT-3.5 Turbo (ablation) | TabLaP · WTQ | accuracy: 62.2 -> 76.6 | Helped |
| Add NumSolver instead of replacing it with GPT-3.5 Turbo (ablation) | TabLaP · FTQ | accuracy: 43.5 -> 44.1 | Helped |
| Add trained AnsSelector instead of random branch selection (ablation) | TabLaP · WTQ | accuracy: 68.5 -> 76.6 | Helped |
Surprising findings
- On FTQ, TwAccuracy (53.7%) is much higher than answer accuracy (44.1%), so following trust labels beats blindly trusting answers.
Limitations
- Relies on GPT-3.5 Turbo for NumSolver and Mix-SC, so cost and API dependence are high.
- FTQ is small (1,245 test pairs), limiting statistical power; FTQ TwAccuracy is only moderate.
- Only basic arithmetic and relational operators are handled in the scope of the study.
Experimental setup
Datasets: WikiTableQuestions FeTaQA TabFact
Benchmarks: WikiTableQuestions FTQ TabFact_small
Metrics: accuracy TwAccuracy user regret ratio
Baselines: TAPEX-Large OmniTab-Large GPT-3.5 Turbo GPT-4o Mix-SC Binder DATER Chain-of-Table GPT-4o mini
Builds on: Chain-of-Thought Mix-SC PAL Binder LoRA UCB
Closest papers by meaning
- TableBench: A Comprehensive and Complex Benchmark for Table Question Answering (AAAI 2025 · similarity 0.92)
- Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding (ICLR 2024 · similarity 0.88)
- Template-Driven LLM-Paraphrased Framework for Tabular Math Word Problem Generation (AAAI 2025 · similarity 0.88)
- Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries (AAAI 2024 · similarity 0.88)
- OpenTab: Advancing Large Language Models as Open-domain Table Reasoners (ICLR 2024 · similarity 0.87)
- TACT: Advancing Complex Aggregative Reasoning with Information Extraction Tools (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.