Research map / Vision-language and multimodal / Multimodal LLMs
ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ChemVLM, an open-source chemistry multimodal LLM (InternViT-6B + MLP + ChemLLM-20B) trained in two stages on chemical image-text and exam data, and evaluated on new ChemOCR, MMCR-Bench and MMChemBench benchmarks.
Problem
Existing LLMs and MLLMs cannot handle chemistry tasks that require jointly reasoning over visual chemical information (molecule images, reaction schemes, exam diagrams), and chemical OCR models cannot perform multimodal reasoning.
Approach
Builds a chemistry-specialized MLLM by coupling a ViT vision encoder and a chemistry-pretrained LLM through an MLP projector, trained with a two-stage (alignment then SFT) recipe on curated bilingual chemical multimodal data, plus new chemistry-specific evaluation sets.
Key results
- On ChemOCR, ChemVLM-26B reaches 71.0% average Tanimoto similarity and 42.9% Tanimoto@1.0, far above GPT-4V (15.0% / 2.1%), though below specialized MolScribe (92.0%) and Decimer (85.0%).
- On MMCR-Bench, ChemVLM-26B scores 41.7% versus GPT-4V's 40.1%, and achieves the best CMMU chemistry score (31.6%).
- On MMChemBench, ChemVLM-26B achieves 98.2% on molecule captioning and 80.9% on property prediction, versus GPT-4V's 95.2% and 38.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ChemVLM-26B instead of GPT-4V for chemical OCR | ChemVLM-26B · ChemOCR | Avg Sim. (%): 15.0 vs 71.0 | Helped |
| Use ChemVLM-26B instead of GPT-4V for chemical OCR | ChemVLM-26B · ChemOCR | Tani@1.0 (%): 2.1 vs 42.9 | Helped |
| Use ChemVLM-26B instead of GPT-4V | ChemVLM-26B · ScienceQA Chem | accuracy (%): 71.9 vs 71.2 | No clear effect |
| Train ChemVLM for chemistry-focused multimodal reasoning | ChemVLM-26B · CMMU Chem | accuracy (%): 29.1 vs 31.6 | Helped |
| Use ChemVLM-26B instead of GPT-4V | ChemVLM-26B · MMCR-Bench | accuracy (%): 40.1 vs 41.7 | Helped |
| Add visual modality to chemical molecule understanding | ChemVLM-26B · MMChemBench | molecule caption (%): ChemLLM 92.6; ChemVLM-26B 98.2 | Helped |
| Add visual modality to chemical molecule understanding | ChemVLM-26B · MMChemBench | property prediction (%): ChemLLM 72.2; ChemVLM-26B 80.9 | Helped |
Surprising findings
- Specialized OCR models (MolScribe, Decimer) remain well ahead of ChemVLM on chemical OCR, which the authors acknowledge.
- ChemVLM scores slightly below GPT-4V on the chemistry part of ScienceQA (71.2 vs 71.9), which the authors attribute to common-sense questions.
Limitations
- Only image modality is supported; no molecular graph or time-series input.
- High computational cost may hinder wider adoption.
- Chemical OCR accuracy lags specialized models such as MolScribe and Decimer.
Experimental setup
Datasets: DECIMER HDM MolScribe dataset PEACE USPTO-50K OpenDataLab chemistry exam questions ChemBench CMMU ScienceQA Scibench RanDepict-augmented molecule images
Benchmarks: ChemOCR MMCR-Bench MMChemBench CMMU ScienceQA Scibench
Metrics: Tanimoto similarity Tanimoto@1.0 accuracy score (total points)
Baselines: Decimer MolScribe Qwen-VL-Chat LLaVA-v1.5-13B InternVL-v1.5 Yi-VL-Plus GPT-4V ChemLLM GPT-4
Builds on: LLaVA InternVL ChemLLM ChemBench RanDepict
Closest papers by meaning
- TinyChemVL: Advancing Chemical Vision-Language Models via Efficient Visual Token Reduction and Complex Reaction Tasks (AAAI 2026 · similarity 0.91)
- Omni-Mol: Multitask Molecular Model for Any-to-any Modalities (NeurIPS 2025 · similarity 0.89)
- KnowMol: Advancing Molecular Large Language Models with Multi-Level Chemical Knowledge (NeurIPS 2025 · similarity 0.89)
- Can LLMs Solve Molecule Puzzles? A Multimodal Benchmark for Molecular Structure Elucidation (NeurIPS 2024 · similarity 0.87)
- Mol-LLaMA: Towards General Understanding of Molecules in Large Molecular Language Model (NeurIPS 2025 · similarity 0.87)
- Structured Chemistry Reasoning with Large Language Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.