Research map / Vision-language and multimodal / Multimodal LLMs
Primitive Vision: Improving Diagram Understanding in MLLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PRIMITIVE, which adds a geometry-grounded vision encoder (GeoGLIP, extended from GLIP) and a feature router that turns hierarchical feature maps into soft visual prompts fused with CLIP tokens for LLMs, improving math diagram reasoning.
Problem
MLLMs misperceive geometric primitives (lines, circles, angles, junctions, boundaries) in mathematical diagrams, and CLIP-style encoders handle diagrams poorly, limiting visual mathematical reasoning.
Approach
Train an auxiliary GLIP-based encoder (GeoGLIP) on synthetic diagrams with box, shape, junction and boundary annotations, then use a soft feature router over its pyramid features to produce visual soft prompts fused with CLIP tokens for the LLM.
Key results
- PRIMITIVE-Qwen2.5-7B reaches 28.5% on MathVerse testmini, versus 19.3% for LLaVA-NeXT (8B) and 16.6% for G-LLaVA.
- On MathVista testmini, PRIMITIVE-Qwen2.5-7B scores 50.4% overall, versus 37.4% for PRIMITIVE-7B with LLaMA2-7B and 36.7% for SPHINX-Plus-13B.
- On GeoQA, PRIMITIVE-Qwen2.5-7B achieves 79.6% top-1 accuracy, compared with 66.7% for MAVIS-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add geometric soft visual prompts (ablation) | PRIMITIVE with DeepSeek-math-7B · MathVista testmini | accuracy: 42.3 -> 48.7 | Helped |
| add geometric soft visual prompts (ablation) | PRIMITIVE with Qwen2.5-math-7B · MathVista testmini | accuracy: 44.0 -> 50.4 | Helped |
| add geometric soft visual prompts (ablation) | Qwen2.5-VL-7B · MathVista and MathVerse | accuracy: 65.0 -> 69.7; 45.3 -> 51.0 | Helped |
| replace GLIP with math-fine-tuned GeoGLIP (ablation) | PRIMITIVE · GeoQA | top-1 accuracy: 65.3 -> 67.0 | Helped |
| remove the CLIP visual encoder (ablation) | PRIMITIVE · GeoQA | top-1 accuracy: 67.0 -> 66.1 | Hurt |
| remove GeoGLIP soft prompts (ablation) | PRIMITIVE · GeoQA | top-1 accuracy: 67.0 -> 64.2 | Hurt |
| use a soft router instead of constant weights (ablation) | PRIMITIVE · GeoQA | top-1 accuracy: 62.8 -> 67.0 | Helped |
| use two sequential projectors in channel-wise fusion (ablation) | PRIMITIVE · GeoQA | top-1 accuracy: 66.98% -> 64.32% (-2.66%) | Hurt |
| use separate projectors rather than shared parameters in sequence-wise fusion (ablation) | PRIMITIVE · GeoQA | top-1 accuracy: 64.32% -> 66.58% (+2.26%) | Helped |
| add Gaussian noise to evaluation images (ablation) | PRIMITIVE · GeoQA | top-1 accuracy: 67.0 -> 65.7 | Hurt |
| add Gaussian noise to evaluation images (ablation) | PRIMITIVE (-) · GeoQA | top-1 accuracy: 64.2 -> 61.1 | Hurt |
| correct geometric visual perception errors | GPT-4o on corresponding mathematical questions · Geo170K-derived questions | accuracy: 12% | Helped |
Surprising findings
- Providing excessive visual cues (e.g., many bounding boxes and junction coordinates) reduced GPT-4o's top-1 accuracy by 4.2%, and incorrect visual cues caused large drops.
- Direct LoRA fine-tuning of Qwen2.5-VL-7B on math data slightly reduced its performance, while adding soft visual prompts yielded gains.
Limitations
- GeoGLIP is sensitive to Gaussian noise and dashed lines, which increase false junction detections.
- Gains depend on the LLM backbone; the authors note improvements with DeepSeek-Math and Qwen2.5-Math but the generalist Qwen2.5-VL result is modest (+1.5 on MathVista).
- Relies on off-the-shelf junction and boundary detectors to produce ground truth labels.
Experimental setup
Datasets: Geo170K MathV360K FigureQA synthetic diagrams (Matplotlib-generated) GeoQA
Benchmarks: MathVerse MathVista GeoQA
Metrics: top-1 accuracy
Baselines: G-LLaVA MAVIS-7B Math-LLaVA LLaVA-NeXT SPHINX-Plus GPT-4V LLaVA-1.5 Geoformer UniMath Qwen2.5-VL-7B
Builds on: GLIP LLaVA-1.5 G-LLaVA MAVIS Grounding DINO
Closest papers by meaning
- Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs (ICLR 2026 · similarity 0.90)
- Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency (NeurIPS 2026 · similarity 0.89)
- MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning (NeurIPS 2025 · similarity 0.89)
- MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine (ICLR 2025 · similarity 0.89)
- Do Vision-Language Models Really Understand Visual Language? (ICML 2025 · similarity 0.88)
- DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.