Research map / Vision-language and multimodal / Multimodal LLMs
Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes Qwen2.5-VL-3B and InternVL2-1B on 1.2M arXiv table image-to-LaTeX pairs, then applies GRPO with a dual reward (TEDS-Structure and rendered-image CW-SSIM) named VSGRPO to improve complex-table LaTeX generation.
Problem
Generating compilable, visually faithful LaTeX code from table images is hard for complex tables (large, deeply nested, irregular content), and existing metrics (TEDS, CW-SSIM) do not capture both structure and visual fidelity.
Approach
A dual-reward GRPO (VSGRPO) where the policy is rewarded both by a TEDS-Structure score on LaTeX converted to HTML and by a binarized CW-SSIM score computed on the compiled rendering of the generated LaTeX, applied after SFT on a large arXiv table corpus.
Key results
- Qwen2.5-VL-3B-VSGRPO reaches CW-SSIM 0.6145 on complex tables (+0.0903 over best baseline) and TEDS-Structure 0.9218, the first above 0.9 on complex tables.
- Intern2-VL-1B-VSGRPO attains CW-SSIM 0.8201 on simple tables, surpassing the previous best by 0.049.
- Human evaluation on 200 tables gives Qwen2.5-VL-3B-VSGRPO the most votes in all difficulty levels (42/37/70).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply VSGRPO rather than the strongest baseline | Qwen2.5-VL-3B · Complex tables | CW-SSIM: 0.5242 -> 0.6145 | Helped |
| Apply VSGRPO rather than the strongest baseline | Qwen2.5-VL-3B · Complex tables | TEDS: 0.7448 -> 0.8673 | Helped |
| Apply VSGRPO rather than the strongest baseline | Qwen2.5-VL-3B · Complex tables | TEDS-Structure: 0.8334 -> 0.9218 | Helped |
| Use only complex tables rather than only simple tables for VSGRPO training (ablation) | Qwen2.5-VL-3B-VSGRPO · Complex tables | CW-SSIM: 0.5993 -> 0.6145 | Helped |
| Use only complex tables rather than mixed-complexity data for VSGRPO training (ablation) | Qwen2.5-VL-3B-VSGRPO · Complex tables | TEDS-Structure: 0.9136 -> 0.9218 | Helped |
| Add TEDS-Structure reward (ablation) | Qwen2.5-VL-3B-SFT · Complex tables | TEDS-Structure: 0.9047 -> 0.9155 | Helped |
| Add CW-SSIM reward (ablation) | Qwen2.5-VL-3B-SFT · Complex tables | CW-SSIM: 0.5806 -> 0.6064 | Helped |
| Combine TEDS-Structure and CW-SSIM rewards rather than using CW-SSIM reward only (ablation) | Qwen2.5-VL-3B-GRPO-CW-SSIM · Complex tables | CW-SSIM: 0.6064 -> 0.6145 | Helped |
| Add SFT initialization before VSGRPO rather than training VSGRPO directly from the pretrained model (ablation) | Qwen2.5-VL-3B · Complex tables | TEDS: 0.6884 -> 0.8673 | Helped |
| Add SFT initialization before VSGRPO rather than training VSGRPO directly from the pretrained model (ablation) | Qwen2.5-VL-3B · Complex tables | CW-SSIM: 0.4695 -> 0.6145 | Helped |
Surprising findings
- Without SFT, RL on the pretrained Qwen2.5-VL-3B drops CW-SSIM to 0.4695 and TEDS to 0.6884, showing SFT is essential before RL.
- Intern2.5-VL-78B has a compile ratio of only 0.4515 on complex tables despite its size.
Limitations
- Rendering LaTeX for CW-SSIM creates a costly training bottleneck, limiting RL to 5,936 complex tables.
- No error bars or repeated runs due to roughly one-week training time per experiment.
- Evaluation relies on the authors' own test split and a single external benchmark, mostly simple tables.
Experimental setup
Datasets: arXiv LaTeX table corpus (1,209,986 pairs) arXiv 2024 test set (496 simple, 354 medium, 361 complex) external benchmark from LATTE paper
Benchmarks: arXiv 2024 test set (simple/medium/complex) external LATTE benchmark
Metrics: CW-SSIM compile ratio TEDS TEDS-Structure human preference votes
Baselines: Mathpix GPT-4o Qwen2.5-VL-72B InternVL2.5-78B Nougat LATTE
Builds on: GRPO PPO Nougat LATTE VLM-R1
Closest papers by meaning
- TableMix: Enhancing Multimodal Table Reasoning in MLLMs from a Data-Centric Perspective (CVPR 2026 · similarity 0.88)
- Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generation (ICLR 2026 · similarity 0.87)
- VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models (ICLR 2026 · similarity 0.87)
- SynTab-LLaVA: Enhancing Multimodal Table Understanding with Decoupled Synthesis (CVPR 2025 · similarity 0.87)
- Seeing is Improving: Visual Feedback for Iterative Text Layout Refinement (CVPR 2026 · similarity 0.86)
- Multimodal Tabular Reasoning with Privileged Structured Information (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.