Research map / Large language models / Language, safety and interpretability
NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NeedleInATable, a long-context benchmark treating each table cell as a needle (cell-locating and cell-lookup queries), evaluates LLMs/MLLMs, and shows that fine-tuning on GPT-4o-synthesized CoT NIAT data improves downstream table tasks.
Problem
Existing long-context benchmarks focus on unstructured text and tabular benchmarks emphasize high-level reasoning, leaving LLMs' fine-grained cell-level perception in long structured tables under-evaluated.
Approach
Treat every table cell as a needle and test models on locating a cell by row/column index or retrieving it via a lookup question, across flat, horizontal and hierarchical tables in Markdown, HTML and image formats; then use GPT-4o-generated chain-of-thought NIAT training data (strong2weak synthesis) to fine-tune weaker models, which also improves downstream table tasks.
Key results
- Fine-tuning Llama3.1-8B-Instruct and Qwen2.5-7B-Instruct on 12K synthesized NIAT data improves average downstream accuracy by 14.55% (Llama3.1-8B: 48.40 to 65.39), and NIAT average by 3.78%.
- DeepSeek-R1 achieves the best NIAT overall score (73.45) among text models in the main table, while open LLMs obtain much higher cell-lookup than cell-locating accuracy (e.g., Qwen3-30B-A3B 78.62 lookup vs 16.49 locating).
- Across MLLMs, GLM-4.1V-Thinking-9B reaches 25.85 cell-locating and 75.12 cell-lookup averages, while cell-locating remains far below lookup for all models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-7B-Instruct on synthesized NIAT data | Qwen2.5-7B-Instruct · Four downstream benchmarks, average accuracy | average accuracy: 51.96 -> 64.06 | Helped |
| Fine-tune Llama3.1-8B-Instruct on synthesized NIAT data | Llama3.1-8B-Instruct · Four downstream benchmarks, average accuracy | average accuracy: 48.40 -> 65.39 | Helped |
| Fine-tune Qwen2.5-7B-Instruct on synthesized NIAT data | Qwen2.5-7B-Instruct · TabFact | exact match accuracy: 70.00 -> 61.28 | Hurt |
| Fine-tune Llama3.1-8B-Instruct on synthesized NIAT data | Llama3.1-8B-Instruct · Four downstream benchmarks | exact match accuracy: 49.90 -> 67.43 WTQ; 62.80 -> 78.57 TabFact; 26.10 -> 49.41 HiTab; 54.78 -> 66.15 TABMWP | Helped |
| Add cell-lookup training alongside cell-locating training (ablation) | Llama3.1-8B-Instruct fine-tuned on cell-locating data · TabFact | exact match accuracy: 67.45 -> 78.57 | Helped |
| Add cell-lookup training alongside cell-locating training (ablation) | Llama3.1-8B-Instruct fine-tuned on cell-locating data · HiTab | exact match accuracy: 33.44 -> 49.41 | Helped |
| Add cell-lookup training alongside cell-locating training (ablation) | Llama3.1-8B-Instruct fine-tuned on cell-locating data · TABMWP | exact match accuracy: 70.50 -> 66.15 | Mixed |
| Add cell-locating training alongside cell-lookup training (ablation) | Llama3.1-8B-Instruct fine-tuned on cell-lookup data · TabFact | exact match accuracy: 53.50 -> 78.57 | Helped |
| Add cell-locating training alongside cell-lookup training (ablation) | Llama3.1-8B-Instruct fine-tuned on cell-lookup data · HiTab | exact match accuracy: 35.00 -> 49.41 | Helped |
| Add cell-locating training alongside cell-lookup training (ablation) | Llama3.1-8B-Instruct fine-tuned on cell-lookup data · TABMWP | exact match accuracy: 69.44 -> 66.15 | Hurt |
| Use synthesized NIAT training rather than fine-tuning on four downstream datasets (ablation) | Llama3.1-8B-Instruct · WTQ, TabFact, HiTab, and TABMWP | exact match accuracy: 64.78 -> 67.43 WTQ; 78.13 -> 78.57 TabFact; 48.22 -> 49.41 HiTab; 81.79 -> 66.15 TABMWP | Mixed |
| Increase cropped table size | LLMs and MLLMs on NIAT cell-locating · Cropped tables from 8×8 to 32×32 | accuracy | Hurt |
Surprising findings
- Cell-locating accuracy is far below cell-lookup accuracy for nearly all models, suggesting LLMs retrieve cells via semantic matching rather than true two-dimensional structure understanding.
- Performance gap between NIAT and downstream tasks suggests downstream benchmark results may rely on dataset-specific shortcuts or data leakage.
Limitations
- Benchmark uses only single English tables from public datasets, which are mostly medium-sized; no multi-table or multilingual coverage.
- Does not cover hybrid document content combining tables, text and images.
- Tables are mostly small-to-medium and average length is modest, so very long tables with thousands of rows are not tested.
Experimental setup
Datasets: WTQ HiTab AIT-QA TabFact TABMWP
Benchmarks: NeedleInATable WTQ TabFact HiTab TABMWP
Metrics: exact match accuracy
Baselines: Mistral-7B-Instruct-v0.3 Llama3.1-8B-Instruct Qwen2.5-7B-Instruct StructLLM TableGPT2 TableLLM GPT-4o DeepSeek-R1 QwQ-32B
Builds on: Needle-in-a-Haystack RULER LongBench-v2
Closest papers by meaning
- Multimodal Tabular Reasoning with Privileged Structured Information (NeurIPS 2025 · similarity 0.90)
- Table as a Modality for Large Language Models (NeurIPS 2025 · similarity 0.90)
- TableBench: A Comprehensive and Complex Benchmark for Table Question Answering (AAAI 2025 · similarity 0.89)
- TableDART: Dynamic Adaptive Multi-Modal Routing for Table Understanding (ICLR 2026 · similarity 0.89)
- OpenTab: Advancing Large Language Models as Open-domain Table Reasoners (ICLR 2024 · similarity 0.89)
- TableRAG: Million-Token Table Understanding with Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.