Research map / Vision-language and multimodal / Visual question answering
Compositional Condition Question Answering in Tabular Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COCOTAB, an MLLM for table image QA that adds row/column patch tokens and question-conditioned visual tokens to LLaVA-style models, and introduces the MMTU benchmark with 8921 QA pairs.
Problem
MLLMs struggle with table image question answering, especially compositional-condition queries, due to patch-based vision encoders misaligning rows and columns and to models overlooking question conditions.
Approach
Augment ViT patches with full-row and full-column patches using separate projections, and add cross-attention so visual tokens are re-weighted by question embeddings (conditional tokens), within a LLaVA-style architecture with Qwen2 as LLM.
Key results
- COCOTAB reaches 0.43 accuracy on MMTU compositional-condition (CC) tasks versus 0.28 for the best open-source baseline (Monkey and TextMonkey), and 0.38 on calculation/reasoning (CR).
- On WTQ, COCOTAB scores 0.45 versus 0.33 for DocOwl, the best prior baseline in the table.
- Ablation on MMTU shows both row/column patches and conditional tokens improve over the LLaVA baseline, with the combination performing best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COCOTAB instead of DocOwl | Tabular question answering models · MMTU IE | accuracy: 0.65 -> 0.68 | Helped |
| Use COCOTAB instead of DocOwl | Tabular question answering models · MMTU RC | accuracy: 0.45 -> 0.50 | Helped |
| Use COCOTAB instead of Monkey | Tabular question answering models · MMTU CC | accuracy: 0.28 -> 0.43 | Helped |
| Use COCOTAB instead of DocOwl | Tabular question answering models · MMTU CR | accuracy: 0.07 -> 0.38 | Helped |
| Use COCOTAB instead of DocOwl | Tabular question answering models · WTQ | accuracy: 0.33 -> 0.45 | Helped |
| Use COCOTAB instead of DocOwl | Tabular question answering models · TabFact | accuracy: 0.61 -> 0.74 | Helped |
| Use COCOTAB instead of DocOwl | Tabular question answering models · ComTQA | accuracy: 0.31 -> 0.34 | Helped |
| Add row and column patches (ablation) | LLaVA on MMTU · MMTU | accuracy | Helped |
| Add conditional tokens to attend to the image based on the question (ablation) | LLaVA on MMTU · MMTU | accuracy | Helped |
Surprising findings
- GPT-4o reaches only 0.47 on compositional-condition questions despite 0.88 on individual-element questions.
- Scaling LLaVA from 7B to 13B yields no significant gain on CC and CR tasks, which the authors attribute to the visual encoder rather than the LLM.
Limitations
- Evaluation of open-ended answers relies on an LLM judge (Qwen2.5-72B-Instruct), which the authors note is not fully accurate.
- Analysis and the CC/CR comparison rely on a small sample of 60 questions per category from WTQ.
- Training data for stage 2 requires GPT-4 generation and manual calibration, limiting scalability.
Experimental setup
Datasets: WTQ TabFact ComTQA Pub1M NAT-QA PlotQA OCR-VQA LAION CC12M
Benchmarks: MMTU WTQ TabFact ComTQA
Metrics: accuracy
Baselines: LLaVA-1.6-7B LLaVA-1.6-13B Monkey TextMonkey mPlug-Owl DocOwl ShareGPT4V VisCPM InstructBLIP Donut
Builds on: LLaVA SigLIP Qwen2
Closest papers by meaning
- Multimodal Tabular Reasoning with Privileged Structured Information (NeurIPS 2025 · similarity 0.87)
- TableBench: A Comprehensive and Complex Benchmark for Table Question Answering (AAAI 2025 · similarity 0.87)
- SynTab-LLaVA: Enhancing Multimodal Table Understanding with Decoupled Synthesis (CVPR 2025 · similarity 0.86)
- ConTextTab: A Semantics-Aware Tabular In-Context Learner (NeurIPS 2025 · similarity 0.86)
- TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusing (AAAI 2026 · similarity 0.86)
- Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.