Research map / Vision-language and multimodal / CLIP and zero-shot
MolVision: Molecular Property Prediction with Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MolVision, a benchmark of ten molecular datasets pairing RDKit skeletal images with SMILES/SELFIES, evaluating nine VLMs zero-shot, few-shot and LoRA-finetuned, plus a Tanimoto-guided contrastive adaptation of the vision encoder.
Problem
LLM-based molecular property prediction relies on textual representations (SMILES/SELFIES) that are ambiguous and structurally uninformative; whether visual molecular structure images help vision-language models predict properties is unexplored.
Approach
Augment molecular property datasets with 2D skeletal images rendered by RDKit and feed image plus text prompts to VLMs; adapt the vision encoder with a Tanimoto-similarity-guided NT-Xent contrastive objective alongside LoRA on the LLM decoder.
Key results
- Fine-tuned BLIP-2 with LoRA reaches 0.92 average classification accuracy across five classification datasets, versus 0.85 for XGBoost.
- Tanimoto-guided contrastive vision adaptation (T-Aug) with LoRA improves BACE-V accuracy to 0.91 from 0.86 for LoRA alone and reduces ESOL RMSE from 1.76 to 0.58.
- Few-shot ICL (k=2) GPT-4V averages 0.80 accuracy on classification, the best among evaluated VLMs, while zero-shot performance remains low for most open models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add molecular images to SMILES input (ablation) | BLIP-2 · BACE-V, BBBP-V, HIV-V, ClinTox-V, and Tox21-V | accuracy: BACE-V: 0.71 -> 0.86; BBBP-V: 0.76 -> 0.93; HIV-V: 0.69 -> 0.92; ClinTox-V: 0.64 -> 0.89; Tox21-V: 0.78 -> 0.99 | Helped |
| Add molecular images to SMILES input (ablation) | BLIP-2 · ESOL-V | RMSE: 9.89 -> 1.07 | Helped |
| Add molecular images to SMILES input (ablation) | BLIP-2 · LD50-V | MAE: 7.80 -> 0.49 | Helped |
| Use molecular images without textual input (ablation) | BLIP-2 · BACE-V | accuracy: 0.71 -> 0.15 | Hurt |
| Use molecular images without textual input (ablation) | BLIP-2 · ESOL-V | RMSE: 9.89 -> 32.16 | Hurt |
| Use Tanimoto-similar molecules as contrastive positive pairs (ablation) | BLIP-2 with LoRA · BACE-V | accuracy: 0.86 -> 0.91 | Helped |
| Use Tanimoto-similar molecules as contrastive positive pairs (ablation) | BLIP-2 with LoRA · ESOL-V | RMSE: 1.76 -> 0.58 | Helped |
| Use Tanimoto-similar molecules as contrastive positive pairs (ablation) | BLIP-2 with LoRA · LD50-V | MAE: 0.78 -> 0.10 | Helped |
| Replace Tanimoto-similar contrastive pairs with augmented views of the same molecule (ablation) | BLIP-2 with LoRA · Tox21-V | accuracy: 0.99 -> 0.97 (Aug); 0.99 -> 0.98 (T-Aug) | Mixed |
| Use CoT prompting instead of traditional ICL (ablation) | Janus-Pro 7B · ESOL-V | RMSE: 0.61 -> 0.89 | Hurt |
| Use CoT prompting instead of traditional ICL (ablation) | Janus-Pro 7B · PCQM4M-V | MAE: 0.62 -> 0.38 | Helped |
| Increase in-context examples from k=0 to k=4 (ablation) | CogVLM · ClinTox-V | accuracy: 0.54 to 0.76 | Helped |
Surprising findings
- Image-only inputs perform far worse than text-only inputs (e.g., BLIP-2 BACE-V accuracy 0.15 image-only vs 0.71 text-only), so visual information alone is insufficient.
- Few-shot in-context learning does not consistently improve with more examples; performance often degrades beyond a certain number of examples.
Limitations
- Closed-source models (GPT-4V, GPT-4o) cannot be adapted with LoRA in this study.
- Few-shot setup accepts only one image per prompt, so in-context example images cannot be provided.
- Graph-based representations are not explored.
Experimental setup
Datasets: BACE BBBP HIV ClinTox Tox21 ESOL LD50 QM9 PCQM4Mv2 ChEBI-20
Benchmarks: BACE-V BBBP-V HIV-V ClinTox-V Tox21-V ESOL-V LD50-V QM9-V PCQM4Mv2-V ChEBI-V
Metrics: accuracy F1 score MAE RMSE BLEU-2 BLEU-4 ROUGE-1 ROUGE-2 ROUGE-L METEOR
Baselines: XGBoost RF UniMol MolCA GenRA ChemLLM Galactica-1.3B Galactica-6.7B Davinci-003 MolT5
Builds on: LoRA BLIP-2 NT-Xent ChemLLMBench MolCA
Closest papers by meaning
- TinyChemVL: Advancing Chemical Vision-Language Models via Efficient Visual Token Reduction and Complex Reaction Tasks (AAAI 2026 · similarity 0.88)
- MolTailor: Tailoring Chemical Molecular Representation to Specific Tasks via Text Prompts (AAAI 2024 · similarity 0.88)
- 3D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modeling (ICLR 2025 · similarity 0.88)
- ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area (AAAI 2025 · similarity 0.87)
- Omni-Mol: Multitask Molecular Model for Any-to-any Modalities (NeurIPS 2025 · similarity 0.87)
- Domain-Agnostic Molecular Generation with Chemical Feedback (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.