Research map / Label-efficient and robust learning / OOD and anomaly detection
Debiasing Multimodal Sarcasm Detection with Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DMSD-CL, a debiasing multimodal sarcasm detection framework that combines ChatGPT-based counterfactual augmentation with a re-weighted contrastive loss, and introduces an OOD MSD test set with manually revised samples.
Problem
Multimodal sarcasm detection models over-rely on biased textual words that correlate with labels, so they generalize poorly when word-label associations shift at test time.
Approach
Generate counterfactual samples (sentiment-flipped sarcastic texts, entity-driven sarcastic rewrites of non-sarcastic texts, and synonym-based EDA variants) and train with an adaptive debiasing contrastive loss that repels similar-bias opposite-label samples and attracts dissimilar-bias same-label samples, weighted by cosine similarity.
Key results
- On the OOD test set DMSD-CL reaches 70.25% accuracy and 67.03% F1, versus 37.50% accuracy for the best baseline (HCMKE).
- On the IID test set DMSD-CL reaches 88.95% accuracy and 86.37% F1, slightly above CMGCN (87.55% acc).
- Removing the contrastive module (w/o-Contra) drops OOD accuracy from 70.25% to 25.75%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DMSD-CL debiasing contrastive learning and counterfactual augmentation | multimodal sarcasm detection · MSD IID test | accuracy: 87.55 -> 88.95 | Helped |
| add DMSD-CL debiasing contrastive learning and counterfactual augmentation | multimodal sarcasm detection · MSD OOD test | accuracy: 37.50 -> 70.25 | Helped |
| add image modality (ablation) | DMSD-CL · MSD IID test | accuracy: 87.67 -> 88.95 | Helped |
| add image modality (ablation) | DMSD-CL · MSD OOD test | accuracy: 61.25 -> 70.25 | Helped |
| add text modality (ablation) | DMSD-CL · MSD IID test | accuracy: 67.04 -> 88.95 | Helped |
| add text modality (ablation) | DMSD-CL · MSD OOD test | accuracy: 35.25 -> 70.25 | Helped |
| add adaptive debiasing weighting to contrastive learning (ablation) | DMSD-CL · MSD IID test | accuracy: 88.33 -> 88.95 | Helped |
| add adaptive debiasing weighting to contrastive learning (ablation) | DMSD-CL · MSD OOD test | accuracy: 64.75 -> 70.25 | Helped |
| add contrastive learning with counterfactual samples (ablation) | DMSD-CL · MSD IID test | accuracy: 88.37 -> 88.95 | Helped |
| add contrastive learning with counterfactual samples (ablation) | DMSD-CL · MSD OOD test | accuracy: 25.75 -> 70.25 | Helped |
Surprising findings
- Without contrastive learning the model's OOD accuracy falls to 25.75%, below chance, despite near-equal IID accuracy (88.37%).
Limitations
- OOD test set is small (400 samples) and manually constructed by annotators.
- Relies on closed-source ChatGPT for counterfactual generation, with prompts only in the appendix.
- Baselines are run with open-source code and some results are cited from prior work, so comparisons may be uneven.
Experimental setup
Datasets: MSD multimodal sarcasm dataset (24,635 samples)
Benchmarks: MSD IID test set MSD OOD test set (manually constructed)
Metrics: Accuracy Precision Recall F1 Macro-average F1
Baselines: ResNet ViT TextCNN Bi-LSTM BERT RoBERTa HFM Res-BERT Att-BERT InCrossMGs HCMKE CMGCN
Builds on: SimCLR EDA RoBERTa ViT
Closest papers by meaning
- G^2SAM: Graph-Based Global Semantic Awareness Method for Multimodal Sarcasm Detection (AAAI 2024 · similarity 0.87)
- DocMSU: A Comprehensive Benchmark for Document-Level Multimodal Sarcasm Understanding (AAAI 2024 · similarity 0.86)
- Well, Now We Know! Unveiling Sarcasm: Initiating and Exploring Multimodal Conversations with Reasoning (AAAI 2024 · similarity 0.86)
- Is Sarcasm Detection a Step-by-Step Reasoning Process in Large Language Models? (AAAI 2025 · similarity 0.84)
- SEVADE: Self-Evolving Multi-Agent Analysis with Decoupled Evaluation for Hallucination-Resistant Sarcasm Detection (AAAI 2026 · similarity 0.82)
- Debiased Multimodal Understanding for Human Language Sequences (AAAI 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.