Research map / Large language models / Language, safety and interpretability
Table as a Modality for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAMO, which treats tables as an independent modality by encoding them with a hypergraph transformer and injecting the resulting structure tokens into a decoder-only LLM alongside serialized text, and introduces the StructQA permutation-robustness benchmark.
Problem
Serialized-text table inputs make LLMs lose table structural semantics such as permutation invariance, so answers are unstable under row/column reordering and structure-dependent reasoning suffers.
Approach
Model tables as hypergraphs (leaf cells as nodes, branch cells/headers as hyperedges) with a permutation-invariant set-transformer hypergraph encoder, project the pooled output into the LLM embedding space as learnable structure tokens prepended to the serialized text, and train with next-token prediction.
Key results
- TAMO with SFT on Llama2-7B reaches 71.60 on StructQA vs 62.73 for SFT text-only and 6.47 for TableLlama, and 63.89 on HiTab vs 54.80 SFT.
- In the frozen-LLM setting, TAMO averages a 42.65% relative gain over prompt tuning across the datasets, with +86.06% on HiTab.
- On StructQA, TAMO+SFT (71.60) outperforms GPT-4.1 (60.33) and DeepSeek-R1 (57.47).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add table-structure modality to text-only SFT | Llama2-7B with TAMO+SFT · StructQA | accuracy: 62.73 -> 71.60 | Helped |
| Add table-structure modality to text-only SFT | Llama2-7B with TAMO+SFT · HiTab | accuracy: 54.80 -> 63.89 | Helped |
| Add table-structure modality to text-only SFT | Llama2-7B with TAMO+SFT · WikiTQ | accuracy: 43.28 -> 45.81 | Helped |
| Add table-structure modality to text-only SFT | Llama2-7B with TAMO+SFT · WikiSQL | accuracy: 79.86 -> 85.90 | Helped |
| Add table-structure modality to text-only SFT | Llama2-7B with TAMO+SFT · FeTaQA | BLEU: 37.37 -> 39.01 | Helped |
| Add structural embeddings alongside serialized text (ablation) | TAMO (PT) · StructQA | accuracy: 37.80 -> 59.07 | Helped |
| Add structural embeddings alongside serialized text (ablation) | TAMO (LoRA) · StructQA | accuracy: 45.93 -> 67.67 | Helped |
| Add structural embeddings alongside serialized text (ablation) | TAMO (SFT) · StructQA | accuracy: 62.00 -> 69.40 | Helped |
| Add serialized text alongside structural embeddings (ablation) | TAMO (PT) · StructQA | accuracy: 7.00 -> 59.07 | Helped |
| Add serialized text alongside structural embeddings (ablation) | TAMO (LoRA) · StructQA | accuracy: 7.93 -> 67.67 | Helped |
| Add serialized text alongside structural embeddings (ablation) | TAMO (SFT) · StructQA | accuracy: 6.93 -> 69.40 | Helped |
| Vary the number of table structure tokens among two or more tokens (ablation) | TAMO (frozen LLM) · WikiTQ | accuracy | No clear effect |
Surprising findings
- Graph-only encoder without serialized text fails on generative QA, with accuracy below 8% across settings.
- Encoder trained on one dataset transfers poorly to others; the FetaQA-trained model gives no gains on other datasets.
Limitations
- Relies on pre-structured tables; unstructured-text tables need text-to-table preprocessing.
- Focuses on static single-turn table understanding, not multi-step or multi-turn reasoning.
- Uses a single serialization format; interaction with other serialization templates is not studied.
Experimental setup
Datasets: StructQA HiTab WikiTQ WikiSQL FeTaQA MultiTabQA-geoQuery
Benchmarks: StructQA HiTab WikiTQ WikiSQL FeTaQA MultiTabQA-geoQuery
Metrics: accuracy BLEU robustness under permutation F1
Baselines: TableLlama GPT-3.5 GPT-4 GPT-4.1 DeepSeek-R1 Prompt tuning LoRA CABINET SeaD TAPAS TAPEX
Builds on: HyTrel HyperTrans Set Transformer TableLlama prompt tuning
Closest papers by meaning
- NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables (NeurIPS 2025 · similarity 0.90)
- OpenTab: Advancing Large Language Models as Open-domain Table Reasoners (ICLR 2024 · similarity 0.89)
- Multimodal Tabular Reasoning with Privileged Structured Information (NeurIPS 2025 · similarity 0.88)
- Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.88)
- SynTab-LLaVA: Enhancing Multimodal Table Understanding with Decoupled Synthesis (CVPR 2025 · similarity 0.88)
- TableDART: Dynamic Adaptive Multi-Modal Routing for Table Understanding (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.