Research map / Large language models / Agents, code and math
Are Large Language Models Good Statisticians?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces StatQA, a 11,623-example benchmark (plus 1,163-example mini split) for evaluating LLMs on statistical method applicability and relevant-column selection for hypothesis testing and descriptive statistics, with LoRA fine-tuning and a human comparison.
Problem
LLMs' ability to perform statistical analysis, particularly assessing the applicability of statistical methods (prerequisites, data type, normality, sample size) and selecting relevant columns, is under-explored; existing benchmarks focus on computation accuracy.
Approach
Reverse the usual dataset construction: start from target answers (methods and columns) computed on real tables, run automated prerequisite checks, then synthesize questions from hand-crafted templates, refined by GPT-3.5-Turbo, to build a scalable applicability benchmark; also compare LLMs against human statisticians and non-statisticians.
Key results
- Best non-fine-tuned model GPT-4o with domain knowledge reaches 64.83% overall Acc(C,M) on mini-StatQA.
- Fine-tuned SFT LLaMA-3 8B reaches 77.13% overall, outperforming all in-context learning methods.
- Statistics-background humans in open-book mode reach 53.45%, while closed-book humans reach 18.10-23.28%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaMA-3 8B on the StatQA training set | LLaMA-3 8B · mini-StatQA | Acc(C, M) overall: 36.11 -> 77.13 | Helped |
| Add one example and domain knowledge to the prompt (ablation) | GPT-4o · mini-StatQA | Acc(C, M) overall: 44.23 -> 64.83 | Helped |
| Add one example to the prompt (ablation) | LLaMA-3 8B · mini-StatQA | Acc(C, M) overall: 23.56 -> 31.90 | Helped |
| Add one example and domain knowledge to the prompt (ablation) | LLaMA-2 7B · mini-StatQA | Acc(C, M) overall: 8.08 -> 16.08 | Helped |
| Use chain-of-thought with zero-shot prompting (ablation) | LLaMA-2 7B · mini-StatQA | Acc(C, M) overall: 8.08 -> 6.36 | Hurt |
| Use chain-of-thought with one-shot prompting (ablation) | GPT-4o · mini-StatQA | Acc(C, M) overall: 49.36 -> 48.67 | Hurt |
| Use one-shot prompting instead of zero-shot (ablation) | GPT models · mini-StatQA | Acc(C, M) overall | Helped |
| Use one-shot prompting instead of zero-shot (ablation) | LLaMA-2/3 models · mini-StatQA | Acc(C, M) overall | Helped |
| Increase model scale from Qwen2 1.5B Instruct to Qwen2 72B Instruct | Qwen2 Instruct · mini-StatQA | Acc(C, M) overall: 20.38 -> 47.12 | Helped |
| Provide supplemental applicability information (ablation) | Human (Stats) · mini-StatQA subset | Acc(C, M) overall: 23.28 -> 53.45 | Helped |
Surprising findings
- Chain-of-thought gives minor performance decreases for most LLaMA-2/3 models.
- Humans mainly err on statistical task confusion while LLMs mainly err on applicability, indicating complementary error profiles.
Limitations
- Benchmark covers only five task categories (descriptive statistics and selected hypothesis tests) and 78 source tables.
- Question phrasing is partly synthetic (templates plus GPT-3.5-Turbo refinement) and human experiments involve only 6 participants on 117 examples.
Experimental setup
Datasets: StatQA training split (D_train, from tables 37-78)
Benchmarks: StatQA mini-StatQA
Metrics: Acc(C,M) accuracy on column and method selection error taxonomy proportions
Baselines: GPT-3.5-Turbo GPT-4 GPT-4o LLaMA-2 7B LLaMA-2 13B LLaMA-3 8B LLaMA-3 8B Instruct
Builds on: LoRA chain-of-thought prompting LLaMA-Factory
Closest papers by meaning
- LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.89)
- Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.89)
- SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines (NeurIPS 2025 · similarity 0.88)
- Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.88)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.88)
- Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.