Research map / Vision-language and multimodal / Multimodal LLMs

MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MechVQA, a 21K-QA benchmark for mechanical drawing understanding across 10 subtasks, and MechVL, a Qwen3-VL-4B model trained with SFT followed by DAPO RL with taxonomy-aligned rewards, which outperforms the strongest closed-source baseline by 7.57 points.

Problem

MLLMs remain brittle on mechanical engineering drawings due to high annotation density, weak domain knowledge, and unreliable spatial/projection reasoning, and no unified benchmark jointly evaluates recognition, reasoning, and judgment on such drawings.

Approach

A semi-automatically constructed benchmark of real part and assembly drawings organized into Recognition, Reasoning, and Judging axes with 10 subtasks, plus a domain-specialized model trained with SFT and then DAPO RL using a composite reward (LLM-judged accuracy, format, and LLM-judged quality).

Key results

• MechVL-4B-RL reaches a total score of 84.85 on MechVQA, +5.94 over the best open-source baseline GLM-4.6V (78.91) and +7.57 over Gemini-3-Pro-Preview (77.28).
• RL after SFT raises the MechVL-4B total from 76.36 (SFT) to 84.85 with targeted self-play RL; DAPO beats GRPO (80.47) and GSPO (78.77) in full-data RL (81.95).
• On hard questions MechVL-4B-RL reaches 75% accuracy versus 66% for the best closed-source baseline Qwen3-VL-Plus.

Surprising findings

• A 4B model fine-tuned on the benchmark's domain data outperforms much larger closed-source models such as GPT-5 and Gemini-3-Pro-Preview on the benchmark.
• Accuracy-only F1 token-overlap reward caused responses to shrink rapidly and performed worst (80.33 total).

Limitations

• Built from public textbooks, handbooks, and design platforms, so it may not reflect proprietary industrial or legacy drawings.
• Focused on 2D drawing understanding; does not address 3D CAD reconstruction or file generation.
• Relies on OCR and LLM-generated QA with partial expert verification; no human upper bound or inter-annotator agreement reported, and contamination cannot be ruled out.

Experimental setup

Datasets: MechVQA

Benchmarks: MechVQA

Metrics: accuracy capability-wise mean total score

Baselines: Qwen3-VL-32B-Instruct GLM-4.6V Gemini-3-Pro-Preview GPT-5 Claude-Sonnet-4.5 Qwen3-VL-Plus GPT-4o InternVL3.5-8B MiniCPM-V-4.5 Mimo-VL-7B-SFT

Builds on: DAPO GRPO GSPO Qwen3-VL

Closest papers by meaning

• VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.85)
• O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model (AAAI 2026 · similarity 0.85)
• VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.85)
• Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing (NeurIPS 2025 · similarity 0.85)
• CAD-GPT: Synthesising CAD Construction Sequence with Spatial Reasoning-Enhanced Multimodal LLMs (AAAI 2025 · similarity 0.85)
• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.85)