Research map / Vision-language and multimodal / Multimodal LLMs

TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TRivia, a self-supervised GRPO-based fine-tuning method that trains VLMs for table recognition from unlabeled table images using attention-guided table QA rewards, yielding the open 3B model TRivia-3B.

Problem

Table recognition (converting table images to HTML/Markdown) relies on costly labeled or teacher-distilled data, and open-source VLMs lag behind proprietary models; the paper asks whether unlabeled table images can improve TR.

Approach

Use table question-answering F1 as a annotation-free reward for GRPO: the TR model's recognized table must let an LLM answer generated questions correctly. Samples are chosen by response-consistency (TEDS among sampled outputs), and QA pairs are generated and filtered via attention-based visual source overlap and cross-checking by a VLM.

Key results

• TRivia-3B reaches 89.88 overall TEDS across four benchmarks, versus 88.93 for Gemini 2.5 Pro and 86.82 for MinerU2.5.
• On CC-OCR, TRivia-3B scores 84.90 TEDS, slightly below Gemini 2.5 Pro (85.56).
• Using Qwen2.5-VL-72B pseudo-labels with SFT drops overall TEDS to 80.02 from the Stage-2 88.57, showing imperfect teacher annotations hurt.

Surprising findings

• Fine-tuning on pseudo-labels from the teacher Qwen2.5-VL-72B degrades performance (SFT -8.37 TEDS on average), even though TRivia-3B trained with QA rewards from the same teacher surpasses it.
• A distilled model trained by SFT on TRivia-3B pseudo-labels slightly outperforms TRivia-3B on CC-OCR (85.84 vs 84.90).

Limitations

• Attention-guided QA generation and cross-checking depend on large teacher and validator models (Qwen2.5-VL-72B, InternVL3-78B), adding cost.
• TRivia-3B does not beat Gemini 2.5 Pro on CC-OCR, and performance on PubTabNet S-TEDS trails expert models.
• Evaluation of the multi-task and noise-robustness claims is preliminary and on small-scale setups.

Experimental setup

Datasets: PubTabNet SynthTabNet MMTab OmniDocBench CC-OCR OCRBench v2 CORD FinTabNet

Benchmarks: OmniDocBench v1.5 CC-OCR OCRBench v2 PubTabNet FinTabNet

Metrics: TEDS S-TEDS

Baselines: SLANNet-plus UniTable InternVL3.5-241B-A30B Qwen2.5-VL-72B Qwen3-VL-235B-A22B Gemini 2.5 Pro GPT-4o GPT-5 dots.ocr DeepSeek-OCR PaddleOCR-VL MinerU2.5

Builds on: GRPO Qwen2.5-VL OTSL TEDS

Closest papers by meaning

• TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy (NeurIPS 2024 · similarity 0.88)
• TDeLTA: A Light-Weight and Robust Table Detection Method Based on Learning Text Arrangement (AAAI 2024 · similarity 0.87)
• TabFlash: Efficient Table Understanding with Progressive Question Conditioning and Token Focusing (AAAI 2026 · similarity 0.86)
• TableBench: A Comprehensive and Complex Benchmark for Table Question Answering (AAAI 2025 · similarity 0.86)
• SynTab-LLaVA: Enhancing Multimodal Table Understanding with Decoupled Synthesis (CVPR 2025 · similarity 0.85)
• Multimodal Tabular Reasoning with Privileged Structured Information (NeurIPS 2025 · similarity 0.85)