Research map / Image generation / Autoregressive and latent generation
Image Over Text: Transforming Formula Recognition Evaluation with Character Detection Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Character Detection Matching (CDM), an image-level formula recognition metric that renders predicted and ground-truth LaTeX, localizes characters by color coding, and matches them with Hungarian matching plus RANSAC consistency checks.
Problem
Text-based formula recognition metrics (BLEU, Edit Distance, ExpRate) are unreliable because the same formula has multiple valid LaTeX representations and scores depend on training-data style, so they unfairly compare models.
Approach
Evaluate formulas at the image level: render both LaTeX strings, color-code each token to localize its bounding box, then match elements via bipartite (Hungarian) matching with token, position, and order costs, filter inconsistent matches with an affine RANSAC check, and report F1 over matched elements (CDM) and ExpRate@CDM.
Key results
- On UniMER-Test, CDM ranks UniMERNet (0.9680) > Mathpix (0.9510) > Texify (0.7550) > Pix2tex (0.6360), with ExpRate@CDM of 0.8110 for UniMERNet.
- In a human preference study over 1,008 Pix2Tex predictions, 64% of votes preferred CDM and 32% judged both credible, reported as 96% consistency with human evaluation.
- Under five GPT-4 rewrites of 50 formulas, CDM stayed at 1 for all samples while BLEU scores were dispersed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CDM instead of BLEU to evaluate formula recognition quality | formula recognition evaluation · Pix2Tex predictions in human preference evaluation | agreement with human judgment: 64% prefer CDM; 32% consider both metrics good | Helped |
| Rewrite formulas in different valid styles while keeping rendered images identical | CDM metric · 50 formulas rewritten five times each | CDM: all samples scoring 1 | Helped |
| Rewrite formulas in different valid styles while keeping rendered images identical | BLEU metric · 50 formulas rewritten five times each | BLEU | Hurt |
| Evaluate model predictions using CDM rather than character-based ExpRate | formula recognition evaluation · UniMER-Test | proportion of completely correct predictions: ExpRate@CDM: Pix2tex 0.2910, Texify 0.4950, Mathpix 0.5000, UniMERNet 0.8110 | Helped |
| Compare Mathpix with UniMERNet using BLEU rather than CDM | formula recognition evaluation · UniMER-Test SCE | model ranking | Hurt |
| Evaluate Pix2Tex on handwriting and screenshot subsets using BLEU | Pix2tex · UniMER-Test HWE and SCE versus SPE and CPE | BLEU | Mixed |
| Use CDM rather than BLEU for formula-level evaluation | GPT-4o · Tiny-Doc-Math formula | model ranking relative to other models: BLEU 0.6431; CDM 0.7330 | Mixed |
| Evaluate formula crops rather than full document screenshots | GPT-4o · Tiny-Doc-Math formula versus document | CDM: 0.7330; 0.6502 | Helped |
| Evaluate formula crops rather than full document screenshots | Mathpix · Tiny-Doc-Math formula versus document | CDM: 0.9480; 0.9567 | Mixed |
| Evaluate document screenshots using CDM | Nougat · Tiny-Doc-Math document | CDM: 0.8326 | Helped |
Surprising findings
- GPT-4o has the highest BLEU but the lowest CDM among the four formula-level models on Tiny-Doc-Math.
- On the SCE subset, BLEU ranks Mathpix above UniMERNet while CDM reverses this ordering, attributed to annotation style bias.
Limitations
- Relies on successful rendering of LaTeX; rendering failures are scored 0.
- Evaluation of document-level outputs depends on regex matching of displayed equations.
- Human evaluation covers only Pix2Tex predictions (1,008 samples) and compares against BLEU only.
Experimental setup
Datasets: UniMER-Test Tiny-Doc-Math
Benchmarks: UniMER-Test Tiny-Doc-Math
Metrics: CDM ExpRate@CDM BLEU ExpRate F1-score
Baselines: BLEU ExpRate Edit Distance Pix2tex Texify Mathpix UniMERNet Nougat GPT-4o
Builds on: DETR bipartite matching ExpRate BLEU Edit Distance
Closest papers by meaning
- LATTE: Improving Latex Recognition for Tables and Formulae with Iterative Refinement (AAAI 2025 · similarity 0.84)
- ASCIIEval: Benchmarking Models' Visual Perception in Text Strings via ASCII Art (ICLR 2026 · similarity 0.82)
- Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition (NeurIPS 2025 · similarity 0.81)
- OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning (NeurIPS 2025 · similarity 0.81)
- General Detection-based Text Line Recognition (NeurIPS 2024 · similarity 0.81)
- MathSpeech: Leveraging Small LMs for Accurate Conversion in Mathematical Speech-to-Formula (AAAI 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.