Research map / Image generation / Autoregressive and latent generation
LATTE: Improving Latex Recognition for Tables and Formulae with Iterative Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LATTE, an iterative-refinement framework for LaTeX recognition of formulae and tables that uses pixel-column diff images ("delta-view") as feedback, a fault localization model, and a refinement model, plus a new table-to-LaTeX dataset.
Problem
Recovering renderable LaTeX source from images of formulae and tables; existing methods generate in a single pass and struggle with complex formulae and with end-to-end table LaTeX.
Approach
Apply iterative refinement to LaTeX recognition, using a Wagner-Fischer-based column-wise image diff (delta-view) between rendered and ground-truth images as feedback, with a fault localization model that predicts the first erroneous token so the refinement model regenerates only the faulty suffix.
Key results
- On IMG2LATEX-100K, LATTE(2) reaches 90.44% Match vs 83.41% for the best prior method (ConvMath), with a 46.08% formula refinement rate.
- On TAB2LATEX, LATTE(2) raises Match from 45.20% to 59.18% after one refinement round; on a 100-sample subset it reaches 67% Match vs 11% for Mathpix and 2% for GPT-4V.
- Delta-view feedback improves table refinement rate from 8.50% (no feedback beyond ground-truth image) to 26.13%, and formula Match from 86.42% to 88.55%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add one iterative refinement round | Latte generation model · Img2LaTeX | Match: 82.27% to 90.44% | Helped |
| Add one iterative refinement round | Latte generation model · Table2LaTeX | Match: 45.20% to 59.18% | Helped |
| Add delta-view feedback (ablation) | Latte refinement without fault localization · Img2LaTeX | Match: 86.42% vs. 88.55% | Helped |
| Add delta-view feedback (ablation) | Latte refinement without fault localization · Table2LaTeX | Match: 49.86% to 59.52% | Helped |
| Add fault localization (ablation) | Latte with delta-view · Img2LaTeX | Match: 88.55% to 90.44% | Helped |
| Add fault localization (ablation) | Latte with delta-view · Table2LaTeX | Match: 59.52% to 59.18% | Hurt |
| Use Latte without fault localization for scripts longer than 512 characters (ablation) | Latte · Img2LaTeX | Match: 90.45 | Helped |
| Use Latte without fault localization for scripts longer than 512 characters (ablation) | Latte · Table2LaTeX | Match: 59.72 | Helped |
Surprising findings
- Fault localization with the full ground-truth image as feedback is less accurate than with delta-view, and the fault localization model slightly lowers table Match (59.52% to 59.18% with the localizer) compared with no localizer.
- GPT-4V and Gemini-1.5-Pro with few-shot prompting achieve very low exact match on both formulae and tables, failing to generate or refine correct LaTeX most of the time.
Limitations
- Only Nougat is explored as the backbone; other MLLMs perform poorly on LaTeX recognition in the authors' experiments.
- Pixel-level Match is harsh and column-wise edit metrics may not reflect human preference; CW-SSIM is added but may be imperfect.
- Fault localization accuracy drops for long formula sources, and table refinement is weak when the fault location is wrong.
Experimental setup
Datasets: IMG2LATEX-100K TAB2LATEX
Benchmarks: IMG2LATEX-100K TAB2LATEX
Metrics: Match CW-SSIM BLEU-4 Edit Refinement rate Time per sample
Baselines: WYGIWYS Double Attention EDPA WAP MI2LaTeX ConvMath Vary-1.8B Llava-v1.5-7B GPT-4V Gemini-1.5-Pro Mathpix
Builds on: Nougat Wagner-Fischer algorithm LoRA
Closest papers by meaning
- Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matching (CVPR 2025 · similarity 0.84)
- Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models (NeurIPS 2025 · similarity 0.83)
- MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024 · similarity 0.81)
- OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations (CVPR 2025 · similarity 0.80)
- TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognition (CVPR 2026 · similarity 0.80)
- Grab What You Need: Rethinking Complex Table Structure Recognition with Flexible Components Deliberation (AAAI 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.