Research map / Vision-language and multimodal / Cross-modal learning
Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MyGO, which tokenizes entity images and text into fine-grained discrete tokens, encodes them with a hierarchical transformer (cross-modal entity encoder, contextual triple encoder, Tucker decoder), and adds a fine-grained contrastive loss for multi-modal knowledge graph completion.
Problem
Existing multi-modal knowledge graph completion (MMKGC) methods compress each modality into a single frozen pre-trained embedding and average multiple images, losing fine-grained semantic detail and cross-modal interactions.
Approach
Treat entity modality data as fine-grained discrete tokens (BEiT visual codes and BERT text tokens, frequency-filtered), feed them with structure embeddings into a transformer entity encoder for full token-level interaction, and augment representations with in-batch contrastive learning over global and local views.
Key results
- On DB15K, MyGO reaches MRR 37.72 and H@1 30.08, versus the best baseline AdaMF at 32.51 MRR (+11.4% MRR relative to the table's second-best row, per the reported improvements).
- On MKG-W, MyGO achieves MRR 36.10 vs MMRNS 35.03; on MKG-Y, MRR 38.44 vs AdaMF 38.06.
- Ablations show removing modality tokenization drops DB15K MRR from 37.72 to 35.48, and removing the contrastive loss drops it to 35.99.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add modality tokenization (MT) (ablation) | MyGO · DB15K | MRR: 35.48 -> 37.72 | Helped |
| Add token refinement (ablation) | MyGO · DB15K | MRR: 36.61 -> 37.72 | Helped |
| Add cross-modal entity encoder (CMEE) (ablation) | MyGO · DB15K | MRR: 34.78 -> 37.72 | Helped |
| Add contextual triple encoder (CTE) (ablation) | MyGO · DB15K | MRR: 34.71 -> 37.72 | Helped |
| Add FGCL contrastive loss (ablation) | MyGO · DB15K | MRR: 35.99 -> 37.72 | Helped |
| Include e_sec in contrastive candidates (ablation) | MyGO FGCL · DB15K | MRR: 36.82 -> 37.72 | Helped |
| Include s(e) in contrastive candidates (ablation) | MyGO FGCL · DB15K | MRR: 37.62 -> 37.72 | Helped |
| Include v(e) in contrastive candidates (ablation) | MyGO FGCL · DB15K | MRR: 37.24 -> 37.72 | Helped |
| Include w(e) in contrastive candidates (ablation) | MyGO FGCL · DB15K | MRR: 37.64 -> 37.72 | Helped |
| Set FGCL loss weight λ to 0.01 rather than 1 (ablation) | MyGO · DB15K | MRR: 37.43 -> 37.72 | Helped |
| Set FGCL loss weight λ to 0.01 rather than 0.1 (ablation) | MyGO · DB15K | MRR: 37.48 -> 37.72 | Helped |
| Set FGCL loss weight λ to 0.01 rather than 0.001 (ablation) | MyGO · DB15K | MRR: 36.91 -> 37.72 | Helped |
Surprising findings
- Removing the refinement step (token frequency filtering) yields MRR 36.61, a smaller drop than removing modality tokenization entirely.
Limitations
- Gains on MKG-W and MKG-Y are small (about +0.9% to +3.1% MRR), and H@10 is not best on some datasets.
- Performance peaks at intermediate token counts, so the token budget m and n needs tuning.
- Cost grows quadratically with token sequence length.
Experimental setup
Datasets: DB15K MKG-W MKG-Y
Benchmarks: DB15K MKG-W MKG-Y
Metrics: MRR Hit@1 Hit@3 Hit@10
Baselines: TransE DistMult ComplEx RotatE TuckER IKRL TBKGC TransAE MMKRL RSME VBKGC OTKGE
Builds on: Tucker SimCSE BEiT BERT
Closest papers by meaning
- Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning (ICLR 2025 · similarity 0.92)
- Tackling Uncertain Correspondences for Multi-Modal Entity Alignment (NeurIPS 2024 · similarity 0.88)
- Beyond Entities: A Large-Scale Multi-Modal Knowledge Graph with Triplet Fact Grounding (AAAI 2024 · similarity 0.87)
- Select and Augment: Enhanced Dense Retrieval Knowledge Graph Augmentation (Abstract Reprint) (AAAI 2024 · similarity 0.87)
- APKGC: Noise-enhanced Multi-Modal Knowledge Graph Completion with Attention Penalty (AAAI 2025 · similarity 0.87)
- A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.