Research map / Vision-language and multimodal / Multimodal LLMs
Multimodal Tabular Reasoning with Privileged Structured Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TURBO, which uses structured markdown tables as privileged training information to generate DeepSeek-R1 reasoning traces, then trains Ovis2-8B via SFT and GRPO to reason over table images.
Problem
Tabular reasoning from table images is hard for MLLMs because clean structured tables, which would help, are usually unavailable at inference time.
Approach
Use structured tables only during training: DeepSeek-R1 generates structure-aware reasoning traces conditioned on the markdown table and ground-truth answer, which are filtered and used to SFT an MLLM, followed by GRPO with rule-based format and accuracy rewards so the model learns to reason from images alone.
Key results
- TURBO reaches 76.42 average accuracy versus 71.28 for the strongest baseline Qwen2.5-VL (reported as +7.2%).
- On MMMU table questions, TURBO scores 57.32 versus 50.61 for Ovis2-CoT.
- Ablation: SFT adds about 10.1% relative average gain over Ovis2 and RL adds about 11.8%, with full TURBO at 13.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with Turbo's structure-aware reasoning supervision and RL | Ovis2-8B · TABMWP, WTQ, HiTab, TAT-QA, TabFact, InfoTabs, and MMMU | accuracy: +7.2% vs. previous SOTA | Helped |
| Add structure-aware reasoning-trace SFT (ablation) | Ovis2 · Evaluation tasks, average | accuracy: +10.1% | Helped |
| Add reward-driven RL (ablation) | Ovis2 · Evaluation tasks, average | accuracy: +11.8% | Helped |
| Add both SFT and RL stages (ablation) | Ovis2 · Evaluation tasks, average | accuracy: +13.8% | Helped |
| Use image-only input instead of structured tables and images (ablation) | HIPPO · TABMWP, WTQ, HiTab, TAT-QA, TabFact, InfoTabs | accuracy: 87.34 -> 85.83; 55.71 -> 49.10; 63.13 -> 57.23; 61.40 -> 56.22; 82.29 -> 80.20; 75.70 -> 72.74 | Hurt |
Surprising findings
- Removing structured tables at inference hurts open MLLMs consistently, motivating the privileged-information setup.
- TURBO slightly surpasses DeepSeek-R1 on InfoTabs (81.89 vs 79.46) despite using only table images at inference.
Limitations
- Structured tables are not used at inference, so the method cannot exploit them when they are available.
- Evaluated mainly on relatively clean regular tables; complex layouts like merged cells in HiTab and MMMU remain challenging.
- A substantial gap remains to proprietary/large models such as DeepSeek.
Experimental setup
Datasets: TABMWP WTQ TAT-QA TabFact InfoTabs
Benchmarks: TABMWP WTQ HiTab TAT-QA FeTaQA TabFact InfoTabs MMMU
Metrics: accuracy average accuracy
Baselines: InternVL-2.5 Qwen2.5-VL MiniCPM-V-2.6 HIPPO Table-LLaVA TabPedia Ovis2 Ovis2-CoT DeepSeek-R1
Builds on: GRPO HIPPO Ovis2 DeepSeek-R1 Chain-of-Table
Closest papers by meaning
- TableDART: Dynamic Adaptive Multi-Modal Routing for Table Understanding (ICLR 2026 · similarity 0.91)
- TableMix: Enhancing Multimodal Table Reasoning in MLLMs from a Data-Centric Perspective (CVPR 2026 · similarity 0.91)
- Decoupling Skeleton and Flesh: Efficient Multimodal Table Reasoning with Disentangled Alignment and Structure-aware Guidance (ICML 2026 · similarity 0.90)
- NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables (NeurIPS 2025 · similarity 0.90)
- ConTextTab: A Semantics-Aware Tabular In-Context Learner (NeurIPS 2025 · similarity 0.89)
- TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.