Research map / Vision-language and multimodal / Multimodal LLMs
MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MAVIS, a rule-based data engine producing diagram-caption (MAVIS-Caption) and visual math instruction (MAVIS-Instruct) datasets, and trains a 7B math MLLM (MAVIS-7B) via four stages including CLIP-Math fine-tuning and DPO.
Problem
MLLMs underperform on visual mathematical problem solving due to poor math diagram encoding by CLIP, diagram-language misalignment, and weak visual chain-of-thought reasoning, with a shortage of large annotated visual math data.
Approach
A fully rule-based, GPT-free data engine renders math diagrams with matching captions and CoT rationales, and a four-stage pipeline (contrastive CLIP-Math tuning, projector alignment, LoRA instruction tuning, DPO on self-generated correct vs incorrect CoTs) trains a math-specialist MLLM.
Key results
- MAVIS-7B reaches the best open-source overall MathVerse score (35.2 CoT-E, 28.4 Acc on testmini), surpassing other 7B models by +9.3% and LLaVA-NeXT (110B) by +6.9%.
- MAVIS-7B scores 68.3 on GeoQA and 50.0 on FunctionQA, versus 67.0 for G-LLaVA on GeoQA.
- MAVIS-7B outperforms Qwen-VL-Max by +6.1% on MMMU-Math and +3.6% on MathVision, and exceeds GPT-4V on the three MathVista math categories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DPO preference alignment (ablation) | MAVIS-7B · MathVerse testmini, All | CoT-E: 33.7 -> 35.2 | Helped |
| add DPO preference alignment (ablation) | MAVIS-7B · MathVerse testmini, All | Acc: 27.5 -> 28.4 | Helped |
| add DPO preference alignment (ablation) | MAVIS-7B · FunctionQA | accuracy: 40.3 -> 50.0 | Helped |
| add DPO preference alignment (ablation) | MAVIS-7B · MMMU-Math | accuracy: 39.2 -> 42.4 | Helped |
| add DPO preference alignment (ablation) | MAVIS-7B · MathVision | accuracy: 18.6 -> 19.2 | Helped |
| add DPO preference alignment (ablation) | MAVIS-7B · MathVista testmini, GPS | accuracy: 63.2 -> 64.1 | Helped |
| add DPO preference alignment (ablation) | MAVIS-7B · MathVista testmini, ALG | accuracy: 58.3 -> 59.2 | Helped |
| add DPO preference alignment (ablation) | MAVIS-7B · MathVista testmini, GEO | accuracy: 63.0 -> 63.2 | Helped |
| add DPO preference alignment (ablation) | MAVIS-7B · We-Math testmini, S1 | accuracy: 56.9 -> 57.2 | Helped |
| add DPO preference alignment (ablation) | MAVIS-7B · We-Math testmini, S2 | accuracy: 37.1 -> 37.9 | Helped |
| add DPO preference alignment (ablation) | MAVIS-7B · We-Math testmini, S3 | accuracy: 33.2 -> 34.6 | Helped |
| add DPO preference alignment (ablation) | MAVIS-7B · GeoQA | accuracy: 66.7 -> 68.3 | Helped |
Surprising findings
- Under the text-lite setting, GPT-4V and Gemini-Pro produce low-quality CoT reasoning, showing MLLMs fail to leverage visual cues.
- DPO gain is more apparent in CoT evaluation than in direct accuracy on MathVerse.
Limitations
- Rule-based diagrams may lack alignment with human aesthetic preferences (3% scored low) and rule-based questions can be rigid in language.
- Some GPT-generated rationales contain reasoning or calculation errors.
- Function and geometry diagram families are limited to a few types (plane geometry, analytic geometry, seven function types).
Experimental setup
Datasets: MAVIS-Caption MAVIS-Instruct Geometry-3K GeoQA+
Benchmarks: MathVerse GeoQA FunctionQA MMMU-Math MathVision MathVista We-Math
Metrics: accuracy CoT evaluation score
Baselines: GPT-4V Gemini-Pro Qwen-VL-Max G-LLaVA Math-LLaVA LLaVA-NeXT InternLM-XComposer2 ShareGPT4V
Builds on: LLaVA CLIP DPO MathVerse Mammoth2
Closest papers by meaning
- Primitive Vision: Improving Diagram Understanding in MLLMs (ICML 2025 · similarity 0.89)
- MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts (CVPR 2025 · similarity 0.89)
- We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning (ICLR 2026 · similarity 0.89)
- MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning (NeurIPS 2025 · similarity 0.89)
- MathSE: Improving Multimodal Mathematical Reasoning via Self-Evolving Iterative Reflection and Reward-Guided Fine-Tuning (AAAI 2026 · similarity 0.88)
- Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.