Research map / Vision-language and multimodal / Cross-modal learning

Tackling Uncertain Correspondences for Multi-Modal Entity Alignment

NeurIPS 2024 · Vision-language and multimodal › Cross-modal learning · approach: Medical multimodal fusion · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TMEA, a multi-modal entity alignment model that handles uncertain correspondences via LLM+in-context attribute alignment, VAE-based missing modality imputation, and cross-attention commonality enhancement with orthogonal constraints.

Problem

Multi-modal entity alignment across MMKGs is hindered by uncertain correspondences: weak inter-modal associations, diverse attribute descriptions, and missing modalities.

Approach

Combines an LLM (GPT-3.5) with in-context learning to align and filter attributes into textual abstracts, imputes missing visual/attribute modalities via VAEs in a shared latent subspace, and enhances inter-modal commonality with cross-attention under orthogonality constraints.

Key results

• TMEA reaches H@1 0.867 and MRR 0.895 on FB15K-DB15K, versus 0.653 H@1 for the strongest prior multi-modal baseline MSNEA.
• On FB15K-YG15K, TMEA achieves H@1 0.818 and H@10 0.916, outperforming MSNEA (H@1 0.443).
• Ablations show removing the MCE module drops H@1 on FB15K-DB15K from 0.867 to 0.749, and removing attribute preprocessing drops it to 0.786.

Measured effects reported in the paper

Surprising findings

• Removing the MMI missing-modality module slightly increases MR on FB15K-DB15K but leaves H@1 nearly unchanged, and the effect varies across datasets.
• EVA performs far worse on FB15K-YG15K (H@1 0.103) than on FB15K-DB15K (0.556), showing dataset sensitivity of baselines.

Limitations

• Attribute encoder relies on a general-purpose LLM (GPT-3.5) rather than KG-specific fine-tuned models.
• Depends on aligned entity annotations for training; performance under scarce labels is not addressed beyond the label-ratio study.
• Frozen ViT and BERT backbones; no end-to-end fine-tuning explored.

Experimental setup

Datasets: FB15K-DB15K FB15K-YG15K

Benchmarks: FB15K-DB15K FB15K-YG15K

Metrics: H@1 H@5 H@10 MR MRR

Baselines: IPTransE GCN-Align BootEA SEA RAC PEEA MixTEA PoE MMEA EVA MSNEA MCLEA

Builds on: TransE MSNEA MMEA BootEA

Closest papers by meaning

• Pseudo-Label Calibration Semi-supervised Multi-Modal Entity Alignment (AAAI 2024 · similarity 0.92)
• Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning (ICLR 2025 · similarity 0.90)
• MESED: A Multi-Modal Entity Set Expansion Dataset with Fine-Grained Semantic Classes and Hard Negative Entities (AAAI 2024 · similarity 0.88)
• Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation (AAAI 2025 · similarity 0.88)
• HLMEA: Unsupervised Entity Alignment Based on Hybrid Language Models (AAAI 2025 · similarity 0.88)
• A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking (AAAI 2024 · similarity 0.86)