Research map / Vision-language and multimodal / Multimodal LLMs

Improving Large Molecular Language Model via Relation-aware Multimodal Collaboration

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CoLLaMo, a LLaMA2-7B-based molecular assistant that fuses 1D SELFIES, 2D graph, and 3D conformer encoders via a relation-aware modality-collaborative projector, and introduces hallucination (CHARM/RCHARM) and GPT-4o-judged caption metrics.

Problem

Large molecular language models often hallucinate and fail to integrate 1D, 2D, and 3D molecular modalities robustly; token-based metrics like BLEU do not capture chemical correctness.

Approach

A multi-level projector with relation-aware collaborative attention that adds shortest-path (2D) and Gaussian-kernel distance (3D) attention biases, unifies 1D/2D/3D tokens via cross-attention, and uses modality dropout and modality embeddings for missing-modality robustness.

Key results

• CoLLaMo achieves 44.5 BLEU and 70.5 METEOR on molecule captioning vs. 38.9/67.1 for LLaMo and 31.1 BLEU for GPT-4o (ICL).
• CHARM hallucination drops to 58.5 for CoLLaMo vs. 64.7 for LLaMo and 98-99 for zero-shot GPT-4/GPT-4o/o1-mini.
• With only 1D input at inference, CoLLaMo keeps captioning BLEU at 28.0 whereas the model without the projector drops to 2.4.

Surprising findings

• Zero-shot GPT-4, GPT-4o, and o1-mini score near-zero BLEU on captioning, far below in-context variants.

Limitations

• Gains on motif counting are not consistently achieved by the multi-modality projector, per the paper's own analysis.
• Evaluation relies on GPT-4o as judge and on BERN2 NER for hallucination, which may carry their own errors.

Experimental setup

Datasets: PubChem molecule-caption pairs GPT-3.5-enriched PubChem captions Mol-Instructions tasks

Benchmarks: molecule captioning IUPAC prediction motif counting descriptive property QA computed property QA CHARM/RCHARM hallucination

Metrics: BLEU METEOR MAE validity CHARM RCHARM LLM score

Baselines: GPT-4 GPT-4o o1-mini LLaMo Mol-Instructions 2D-MoLM 3D-MoLM Uni-Mol Llama2-7B

Builds on: MolCA LLaMo 3D-MoLM Graphormer Uni-Mol

Closest papers by meaning

• LLaMo: Large Language Model-based Molecular Graph Assistant (NeurIPS 2024 · similarity 0.90)
• Omni-Mol: Multitask Molecular Model for Any-to-any Modalities (NeurIPS 2025 · similarity 0.89)
• Hierarchical Graph Tokenization for Molecule-Language Alignment (ICML 2025 · similarity 0.89)
• KnowMol: Advancing Molecular Large Language Models with Multi-Level Chemical Knowledge (NeurIPS 2025 · similarity 0.89)
• 3D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modeling (ICLR 2025 · similarity 0.88)
• Towards 3D Molecule-Text Interpretation in Language Models (ICLR 2024 · similarity 0.88)