Research map / Vision-language and multimodal / Multimodal LLMs

Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uni-MuMER fully fine-tunes Qwen2.5-VL-3B for handwritten math expression recognition with three added data-driven tasks (Tree-CoT, error-driven learning, symbol counting), reaching 79.74% average ExpRate on CROHME.

Problem

Handwritten mathematical expression recognition (HMER) has seen marginal gains from specialized lightweight architectures, and generalist VLMs underperform on it; the paper asks how to inject HMER domain knowledge into a pretrained VLM without changing its architecture.

Approach

Full fine-tuning of a generalist VLM on multiple HMER-specific tasks expressed purely as text outputs: AST-serialized chain-of-thought (Tree-CoT), self-generated error detection/correction (EDL), and symbol counting prepended to LaTeX, trained together with vanilla recognition and no architectural change.

Key results

• Uni-MuMER† (with external data) reaches 79.74% ExpRate and 82.86% ExpRate@CDM averaged across CROHME 2014/2016/2019, versus 55.32% for Gemini2.5-flash and 60.90% for SSAN (zero-shot for VLMs).
• Without external data, Uni-MuMER reaches 73.29% average ExpRate on CROHME, up from 68.64% for the vanilla baseline.
• On HME100K (CJK excluded), Uni-MuMER† achieves 72.66% ExpRate versus 69.50% for TAMER and 64.20% for HiE-VL.

Measured effects reported in the paper

Surprising findings

• CoMER degraded when trained on the large mixed external dataset (46.18% vs 52.29% of CROHME-only setup in the reported table), suggesting lightweight specialized models cannot exploit large diverse data.
• Uni-MuMER-LLAVA mixing general VLM data improved MMMU_val slightly but reduced performance on several HMER test sets, e.g., CROHME-2023 drops from 74.42 to 69.10.

Limitations

• A 400k synthetic subset of MathWriting was left unused due to compute constraints.
• Optimal task mixture ratios and domain mixture ratios were not explored.
• Non-trivial performance drops on out-of-domain datasets (e.g., HME100K leave-one-out drops to 52.38) show limited generalization to very different domains.

Experimental setup

Datasets: CROHME 2014 CROHME 2016 CROHME 2019 CROHME 2023 HME100K MathWriting Im2LaTeXv2

Benchmarks: CROHME 2014 CROHME 2016 CROHME 2019 HME100K CROHME 2023 MathWriting Im2LaTeXv2 MMMU_val Math-Vision MathVista_testmini

Metrics: ExpRate ExpRate@CDM CDM CER EditScore BLEU-4

Baselines: Gemini2.5-flash GPT-4o Doubao-1.5-pro Qwen2.5-VL-3B Qwen2.5-VL-7B Qwen2.5-VL-72B CoMER TAMER SSAN PosFormer HiE-VL VLPG

Builds on: Qwen2.5-VL CoMER CAN LATTE MathNet

Closest papers by meaning

• SSAN: A Symbol Spatial-Aware Network for Handwritten Mathematical Expression Recognition (AAAI 2025 · similarity 0.90)
• TAMER: Tree-Aware Transformer for Handwritten Mathematical Expression Recognition (AAAI 2025 · similarity 0.89)
• Generating Handwritten Mathematical Expressions From Symbol Graphs: An End-to-End Pipeline (CVPR 2024 · similarity 0.87)
• Primitive Vision: Improving Diagram Understanding in MLLMs (ICML 2025 · similarity 0.83)
• MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios (NeurIPS 2025 · similarity 0.83)
• On the Generalization of Handwritten Text Recognition Models (CVPR 2025 · similarity 0.83)