Research map / Graph learning / Graph self-supervised learning
Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MM-GRAPH, a benchmark of seven multimodal (text + image) graph datasets for link prediction, node classification, and knowledge graph completion, and benchmarks GNNs, multimodal GNNs, multimodal KGEs, and feature encoders on it.
Problem
Graph learning benchmarks rarely include visual node features alongside text, so it is unclear how much visual information helps GNNs and knowledge graph embeddings on downstream graph tasks.
Approach
A standardized multimodal graph benchmark with image and text node attributes drawn from Amazon, Goodreads, and CoDEx, plus an empirical study of how feature encoders (CLIP, ImageBind, T5, ViT, DINOv2) and GNN/KGE architectures perform on them.
Key results
- On Amazon-Sports, ImageBind features with SAGE reached MRR 34.32, the best link prediction result among the reported encoder-GNN pairs, and aligned features (CLIP, ImageBind) outperformed unaligned ones.
- Conventional GNNs such as SAGE generally beat multimodal GNNs MMGCN and MGAT, which the authors attribute to per-modality message passing that fuses only at the final stage.
- Using an MLP, multimodal (text + image) features improved over text-only features by more than 6% across datasets and tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multimodal GNNs (MMGCN/MGAT) instead of conventional GNNs | MM-GRAPH link prediction · Amazon-Sports and Amazon-Cloth | MRR | Hurt |
| align image and text embeddings with CLIP or ImageBind instead of using T5 + ViT or T5 + DINOv2 | MM-GRAPH feature encoders · Amazon-Sports and Amazon-Cloth | MRR | Helped |
| use T5 + DINOv2 instead of aligned encoders | MM-GRAPH link prediction · Goodreads-LP | MRR: 45.61 ±0.22 | Helped |
| combine visual and textual features instead of using a single modality | MLP · Amazon-Sports, Amazon-Cloth, Goodreads-LP, Ele-fashion, and Goodreads-NC | MRR or accuracy: 10.76; 18.62; 30.45; 88.73; 72.29 | Helped |
| remove graph structure and use multimodal features alone (ablation) | MLP · Goodreads-NC | accuracy: 58.75 ±0.05 | Hurt |
| use aligned ImageBind features instead of T5 + DINOv2 | MoSE · MM-CoDEx-s | MRR: 40.07 ±0.24; 22.45 ±1.36 | Hurt |
| use ImageBind instead of T5 + DINOv2 | VISTA · MM-CoDEx-s | MRR: 30.39 ±0.65; 26.98 ±1.68 | Helped |
Surprising findings
- Multimodal GNNs (MMGCN, MGAT) do not consistently outperform conventional GNNs such as SAGE.
- On Goodreads-LP, T5 + DINOv2 outperformed CLIP/ImageBind, attributed to OCR capability for text on book covers.
Limitations
- Audio and video modalities are not included due to scarcity of open-source datasets.
- Goodreads-LP uses random negative sampling instead of hard negatives because HeaRT takes over 120 hours.
- Multimodal GNN designs perform poorly, indicating current architectures do not integrate modalities well.
Experimental setup
Datasets: Amazon-Sports Amazon-Cloth Goodreads-LP Goodreads-NC Ele-fashion MM-CoDEx-s MM-CoDEx-m
Benchmarks: MM-GRAPH
Metrics: MRR Hits@1 Hits@10 Accuracy Hits@3
Baselines: GCN SAGE MLP BUDDY MMGCN MGAT MoSE VISTA
Builds on: MMGCN MGAT MoSE VISTA HeaRT
Closest papers by meaning
- Graph4MM: Weaving Multimodal Learning with Structural Information (ICML 2025 · similarity 0.91)
- GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning (CVPR 2026 · similarity 0.91)
- Mario: Multimodal Graph Reasoning with Large Language Models (CVPR 2026 · similarity 0.90)
- No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets (ICML 2025 · similarity 0.89)
- GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs (CVPR 2025 · similarity 0.87)
- InstructG2I: Synthesizing Images from Multimodal Attributed Graphs (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.