Research map / Large language models / Agents, code and math
TACT: Advancing Complex Aggregative Reasoning with Information Extraction Tools
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TACT, a 124-example expert-annotated benchmark of aggregative numerical instructions over InstructIE texts, and an "IE as a tool" few-shot prompting pipeline (table generation, Pandas command generation, execution) that improves LLM accuracy.
Problem
LLMs perform poorly on queries requiring aggregating and computing over information scattered across one or more texts; there is no focused benchmark or decomposed modeling approach for this setting.
Approach
Decompose aggregative text queries into table generation, Pandas command generation, and execution, and treat each step as a few-shot-prompted tool whose outputs feed the final LLM answer or a code interpreter.
Key results
- All evaluated contemporary LLMs score below 38% exact-match accuracy on TACT's 124 examples.
- IE as a tool with Pandas improves GPT-4o from 30.1% (generic) to 42.2% and Claude 3.5 Sonnet to 42.1%, up to about 12 points over baselines.
- Gold table plus Pandas command yields 72-75% for top models (Claude 3.5 Sonnet 74.6%), showing large headroom in the tool outputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| IE as a tool (table + Pandas command tools) vs generic prompting | GPT-4o, Claude 3.5 Sonnet, Gemini models · TACT | exact match accuracy: up to ~12 points (GPT-4o 30.1 to 42.2) | Helped |
| Adding Pandas command generation to the table-tool pipeline (Without Pandas vs With Pandas) (ablation) | IE as a tool across models · TACT | exact match accuracy: small gains, e.g. GPT-4o 41.1 to 42.2 | Helped |
| Gold table and gold Pandas command provided instead of tool outputs (ablation) | Claude 3.5 Sonnet · TACT | exact match accuracy: 42.1 to 74.6 | Helped |
| Chain-of-thought prompting vs generic prompting | Claude 3.5 Sonnet · TACT | exact match accuracy: 28.6 to 37.9 | Helped |
| In-context IE (single-prompt table then Pandas) vs generic prompting | Gemini-Pro · TACT | exact match accuracy: 28.4 generic to 12.3 in-context IE | Hurt |
| Few-shot demonstrations for table generation (ablation) | Gemma-7b and Mistral-7b · TACT table generation | table validity rate: Gemma-7b 0.6% (0-shot) to 25.5% (4-shot); Mistral-7b 1.2% to 34.4% | Helped |
| Few-shot demonstrations for Pandas command generation on generated table (ablation) | Gemma-7b · TACT Pandas command generation | accuracy after execution (generated table): 0 (0-shot) to 1.9 (4-shot) | Helped |
| Using generated tables instead of gold tables for Pandas generation (ablation) | Claude 3.5 Sonnet · TACT Pandas command generation | execution accuracy: 10.1 (gold) vs 9.6 (generated) at 4-shot | Hurt |
Surprising findings
- Pandas command execution accuracy is very low (often below 10%) even for top models, largely due to syntax errors and execution failures.
- Few-shot demonstrations yield little gain for larger models on the end task, and Gemini-Pro's In-context IE score (12.3%) drops well below its CoT score (34.7%).
Limitations
- Sequential tool pipeline propagates errors across table and command stages.
- Method designed for numerical tasks and may not transfer to non-numerical NLP tasks.
- Small benchmark size of 124 examples.
Experimental setup
Datasets: InstructIE TACT
Benchmarks: TACT
Metrics: exact match accuracy semantic similarity (SentenceT5 cosine) ROUGE-L F1 table validity rate execution accuracy
Baselines: Generic prompting Chain-of-Thought In-context IE GPT-4o Claude 3.5 Sonnet Gemini-1.0-Ultra Gemini-1.5-Pro
Builds on: InstructIE chain-of-thought prompting
Closest papers by meaning
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.89)
- ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation (AAAI 2026 · similarity 0.89)
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.89)
- Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Models (ICLR 2026 · similarity 0.88)
- Hypothesis Search: Inductive Reasoning with Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.