Research map / Graph learning / Knowledge and temporal graphs
G^2SAM: Graph-Based Global Semantic Awareness Method for Multimodal Sarcasm Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes G2SAM, a graph-based multimodal sarcasm detector that retrieves k-nearest-neighbor instances in a graph-based semantic space at inference and trains with label-aware graph contrastive learning.
Problem
Multimodal sarcasm detection on social media posts; existing graph-based methods ignore global semantic congruity across existing training instances at inference time.
Approach
A kNN-based inference paradigm over a graph-based semantic space built from fine-grained text-token and image-region graphs fused by a graph-aligned self-attention module, plus a label-aware graph contrastive loss that makes same-label instances more similar so retrieved neighbors share the query's label.
Key results
- G2SAM reaches 90.48% accuracy and 88.48% F1 on the HFM sarcasm dataset, versus 89.59% accuracy and 87.17% F1 for the prior SOTA DIP.
- Ablation: adding kNN inference to FGM raises F1 from 86.43 to 87.35; adding LGCL raises it to 86.92; combining both gives 88.48.
- Plugging G2SAM into DIP improves accuracy from 89.59% to 90.21%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use G2SAM instead of DIP | Multimodal sarcasm detection · Multimodal sarcasm detection test set | Accuracy: 89.59 -> 90.48 | Helped |
| Use G2SAM instead of DIP | Multimodal sarcasm detection · Multimodal sarcasm detection test set | F1: 87.17 -> 88.48 | Helped |
| Add kNN inference to FGM (ablation) | FGM · Multimodal sarcasm detection test set | Accuracy: 89.01 -> 89.86 | Helped |
| Add kNN inference to FGM (ablation) | FGM · Multimodal sarcasm detection test set | F1: 86.43 -> 87.35 | Helped |
| Add LGCL to FGM (ablation) | FGM · Multimodal sarcasm detection test set | Accuracy: 89.01 -> 89.33 | Helped |
| Add LGCL to FGM (ablation) | FGM · Multimodal sarcasm detection test set | F1: 86.43 -> 86.92 | Helped |
| Add LGCL to FGM+kNN (ablation) | FGM+kNN · Multimodal sarcasm detection test set | Accuracy: 89.86 -> 90.48 | Helped |
| Add LGCL to FGM+kNN (ablation) | FGM+kNN · Multimodal sarcasm detection test set | F1: 87.35 -> 88.48 | Helped |
| Add kNN inference to FGM+LGCL (ablation) | FGM+LGCL · Multimodal sarcasm detection test set | Accuracy: 89.33 -> 90.48 | Helped |
| Add kNN inference to FGM+LGCL (ablation) | FGM+LGCL · Multimodal sarcasm detection test set | F1: 86.92 -> 88.48 | Helped |
Surprising findings
- The simple fine-grained graph-aligned model (FGM) alone performs comparably to the more complex MILNet.
Limitations
- Evaluated only on a single public Twitter sarcasm dataset (HFM).
- Hyperparameters such as k=50 and N=2048 were selected by sweeps on the test-set curves reported in the paper.
- Hardware and training compute are not reported.
Experimental setup
Datasets: HFM multimodal sarcasm detection dataset (Twitter)
Benchmarks: HFM multimodal sarcasm detection dataset
Metrics: accuracy precision recall F1 macro-average precision macro-average recall macro-average F1
Baselines: Bi-LSTM BERT ViT Resnet HFM Att-BERT DIP InCrossMGs CMGCN HKEmodel MILNet
Builds on: HFM HKEmodel MILNet CMGCN InCrossMGs
Closest papers by meaning
- Debiasing Multimodal Sarcasm Detection with Contrastive Learning (AAAI 2024 · similarity 0.87)
- DocMSU: A Comprehensive Benchmark for Document-Level Multimodal Sarcasm Understanding (AAAI 2024 · similarity 0.87)
- Well, Now We Know! Unveiling Sarcasm: Initiating and Exploring Multimodal Conversations with Reasoning (AAAI 2024 · similarity 0.85)
- Towards Multimodal Sentiment Analysis via Hierarchical Correlation Modeling with Semantic Distribution Constraints (AAAI 2025 · similarity 0.85)
- Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum (AAAI 2025 · similarity 0.84)
- Adaptive Graph Learning for Multimodal Conversational Emotion Detection (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.