Research map / Vision-language and multimodal / Cross-modal learning

Graph4MM: Weaving Multimodal Learning with Structural Information

ICML 2025 · Vision-language and multimodal › Cross-modal learning · approach: Medical multimodal fusion · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Graph4MM, which injects multi-hop graph structure into a frozen-LLM multimodal pipeline via Hop-Diffused/Hop-Aware Attention and an MM-QFormer, and argues graphs should guide fusion rather than be treated as a separate modality.

Problem

Multimodal data have many-to-many relations (text and images linked through graph structure), but existing VLMs model only one-to-one image-text pairs and prior multimodal graph learning (MMGL) treats the graph as a standalone modality and ignores hop distance.

Approach

Use multi-hop graph connectivity to guide intra-modal self-attention (causal masking plus PPR-style hop diffusion, or lightweight learned hop embeddings) and fuse text and image tokens with a multi-mapping querying transformer (MM-QFormer), rather than encoding the graph as an extra modality.

Key results

• Hop-Diffused MM-QFormer on OPT-125M reaches ROUGE-L 0.4076 and CIDEr 0.7831 on the generative task, above MMGL's best subgraph text+image setting (0.4041 / 0.7712).
• On Ele-Fashion with OPT-125M, Hop-Diffused MM-QFormer achieves 100% accuracy, versus 99.85% for the best MMGL baseline.
• The paper reports an average improvement of 6.93% across textual and visual metrics.

Measured effects reported in the paper

Surprising findings

• Adding GCN graph embeddings as a standalone modality (MMGL's Subgraph's T&I + GNN) often degrades performance relative to using text and image attributes alone.
• Pretrained VLMs (BLIP-2, Qwen2-VL-7B) perform worst, often generating image descriptions instead of answering the instruction.

Limitations

• Gains on the discriminative task are near ceiling (about 100% accuracy), limiting the discriminative comparison.
• Experiments are limited to two datasets with small backbones (OPT-125M, LLaMA-1B), so scaling is untested.
• Graph subgraph sampling is heuristic (e.g., sampling 2 second-hop neighbors), which may affect results.

Experimental setup

Datasets: WikiWeb2M Ele-Fashion

Benchmarks: WikiWeb2M Ele-Fashion

Metrics: BLEU-4 ROUGE-L CIDEr accuracy recall precision

Baselines: OPT-125M LLaMA-1B BLIP-2 (OPT-2.7B) Qwen2-VL-7B-Instruct MMGL

Builds on: MMGL BLIP-2 Personalized PageRank GAT

Closest papers by meaning

• Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learning (CVPR 2025 · similarity 0.91)
• Mario: Multimodal Graph Reasoning with Large Language Models (CVPR 2026 · similarity 0.89)
• GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning (CVPR 2026 · similarity 0.88)
• GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs (CVPR 2025 · similarity 0.87)
• Structure-Aware Multimodal Sequential Learning for Visual Dialog (AAAI 2024 · similarity 0.86)
• On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.86)