Research map / Vision-language and multimodal / Cross-modal learning
Multi-Granular Multimodal Clue Fusion for Meme Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MGMCF, a multi-granular multimodal clue fusion model with object-level visual features, a global-local cross-modal attention interaction, and a dual-semantic contrastive loss, evaluated on the MET-MEME bilingual meme dataset for four MMU tasks.
Problem
Multimodal meme understanding (metaphor recognition, sentiment, intention, offensiveness) suffers from losing fine-grained object-level metaphorical visual clues and from weak correlation between meme text and image that naive fusion fails to handle.
Approach
Combines image-level VGG16 features with object-level region features, and fuses text and image via a global multimodal context token that bidirectionally cross-attends with local unimodal features (CAP), plus unimodal prediction heads and a contrastive dual-semantic loss.
Key results
- On MET-MEME English, precision on offensiveness detection improves by 8.14% over the SoTA (M3F).
- Accuracy improves by 3.53% (metaphor recognition), 3.89% (sentiment analysis), and 3.52% (intention detection) on English.
- Removing the global-local interaction causes the largest drop in ablations, e.g. OD precision falls by 5.16%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MGMCF instead of M3F_add | MET-MEME English sentiment analysis · MET-MEME English sentiment analysis | accuracy: 30.47 -> 34.36 | Helped |
| Use MGMCF instead of M3F_add | MET-MEME Chinese sentiment analysis · MET-MEME Chinese sentiment analysis | accuracy: 39.95 -> 42.11 | Helped |
| Use MGMCF instead of M3F_add | MET-MEME English intention detection · MET-MEME English intention detection | accuracy: 44.40 -> 47.92 | Helped |
| Use MGMCF instead of M3F_add | MET-MEME Chinese intention detection · MET-MEME Chinese intention detection | accuracy: 55.25 -> 58.56 | Helped |
| Use MGMCF instead of M3F_add | MET-MEME English offensiveness detection · MET-MEME English offensiveness detection | accuracy: 76.17 -> 78.11 | Helped |
| Use MGMCF instead of M3F_add | MET-MEME Chinese offensiveness detection · MET-MEME Chinese offensiveness detection | accuracy: 80.81 -> 82.15 | Helped |
| Use MGMCF instead of M3F_add | MET-MEME English metaphor recognition · MET-MEME English metaphor recognition | accuracy: 83.98 -> 87.51 | Helped |
| Use MGMCF instead of M3F_add | MET-MEME Chinese metaphor recognition · MET-MEME Chinese metaphor recognition | accuracy: 77.01 -> 78.39 | Helped |
| Add object-level semantic mining (ablation) | MGMCF · MET-MEME English sentiment analysis | accuracy: 32.47 -> 34.36 | Helped |
| Add unimodal prediction (ablation) | MGMCF · MET-MEME English sentiment analysis | accuracy: 32.85 -> 34.36 | Helped |
| Add global-local interaction (ablation) | MGMCF · MET-MEME English sentiment analysis | accuracy: 31.76 -> 34.36 | Helped |
| Add dual-semantic guided strategy (ablation) | MGMCF · MET-MEME English sentiment analysis | accuracy: 33.59 -> 34.36 | Helped |
Surprising findings
- Naive MET-style fusion can underperform unimodal DenseNet-161 and ResNet-50 on offensiveness detection.
Limitations
- Improvements are reported only on MET-MEME, a small bilingual dataset.
- Hardware and training cost details are not reported.
Experimental setup
Datasets: MET-MEME
Benchmarks: MET-MEME
Metrics: accuracy weighted precision weighted recall
Baselines: VGG16 DenseNet-161 ResNet-50 Multi-BERT_EfficientNet Multi-BERT_ViT Multi-BERT_PiT MET_add MET_cat M3F_add M3F_cat
Builds on: M3F MET-MEME Multilingual BERT VGG16
Closest papers by meaning
- CAMEL: Capturing Metaphorical Alignment with Context Disentangling for Multimodal Emotion Recognition (AAAI 2024 · similarity 0.86)
- Debiased Multimodal Understanding for Human Language Sequences (AAAI 2025 · similarity 0.85)
- DocMSU: A Comprehensive Benchmark for Document-Level Multimodal Sarcasm Understanding (AAAI 2024 · similarity 0.85)
- Hateful Meme Detection through Context-Sensitive Prompting and Fine-Grained Labeling (Student Abstract) (AAAI 2025 · similarity 0.85)
- TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection (AAAI 2026 · similarity 0.84)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.