Research map / Vision-language and multimodal / Human and embodied modeling
DocMSU: A Comprehensive Benchmark for Document-Level Multimodal Sarcasm Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DocMSU, a 102,588-sample document-level news benchmark of text-image pairs for multimodal sarcasm detection and localization, plus a Swin-Transformer-based fine-grained text-pixel fusion baseline.
Problem
Existing multimodal sarcasm understanding benchmarks and methods are sentence-level (tweets, short utterances), failing to capture sparse sarcasm clues spread across long news documents and images.
Approach
Reshape document token representations into an L×L matrix and add it to sliding windows of ResNet pixel features, fusing them with Swin-Transformer shifted-window attention to align word-level text cues with pixel-level image cues; new relaxed localization metrics EM50/EM70 and BitError are introduced.
Key results
- Ours (Swin-Base) reaches 97.83% accuracy and 87.25 F1 on sarcasm detection, versus 96.19% for CLIP and 93.15% for ViLT.
- For textual sarcasm localization, Ours (Swin-Base) achieves 52.19 EM50 versus 39.34 for CLIP and 44.99 for MSTI.
- Fusing text and image raises detection accuracy from 87.12% (text only) to 97.83%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed fine-grained image-text fusion instead of feature concatenation (ablation) | DocMSU sarcasm detection · DocMSU | Accuracy: 90.27 -> 97.83 | Helped |
| Use the proposed fine-grained image-text fusion instead of feature concatenation (ablation) | DocMSU image sarcasm localization · DocMSU | AP50: 21.51 -> 41.04 | Helped |
| Use the proposed fine-grained image-text fusion instead of feature concatenation (ablation) | DocMSU text sarcasm localization · DocMSU | EM50: 43.16 -> 52.19 | Helped |
| Add image modality to text (ablation) | DocMSU sarcasm detection · DocMSU | Accuracy: 87.12 -> 97.83 | Helped |
| Add text modality to image (ablation) | DocMSU sarcasm detection · DocMSU | Accuracy: 74.83 -> 97.83 | Helped |
| Add text modality to image (ablation) | DocMSU image sarcasm localization · DocMSU | AP50: 21.78 -> 35.29 | Helped |
| Add image modality to text (ablation) | DocMSU text sarcasm localization · DocMSU | EM50: 45.86 -> 49.74 | Helped |
| Use Swin-Transformer Base instead of Tiny (ablation) | Ours sarcasm detection · DocMSU | Accuracy: 96.40 -> 97.83 | Helped |
| Use Swin-Transformer Base instead of Small (ablation) | Ours sarcasm detection · DocMSU | Accuracy: 96.82 -> 97.83 | Helped |
| Use the proposed method instead of CLIP | DocMSU sarcasm detection · DocMSU | Accuracy: 96.19 -> 97.83 | Helped |
| Use the proposed method instead of ViLT | DocMSU image sarcasm localization · DocMSU | AP50: 30.73 -> 35.29 | Helped |
| Use the proposed method instead of BERT-base | DocMSU text sarcasm localization · DocMSU | EM50: 45.86 -> 52.19 | Helped |
Surprising findings
- The text-only BERT-base baseline reaches 87.12% accuracy, showing that text alone carries much of the sarcasm signal in this dataset.
- Image-only Swin-Transformer scores only 74.83% accuracy on detection.
Limitations
- Annotation of sarcasm is subjective; about 5% of hard samples were labeled by a single top-scoring annotator.
- LLMs such as GPT-4 struggle with sarcasm when the text does not obviously signal it.
- Only a small set of baselines and a single fusion design are compared; no analysis of cultural variation.
Experimental setup
Datasets: DocMSU
Benchmarks: DocMSU
Metrics: Accuracy Precision F1-score AP50 AP60 F150 F160 EM50 EM70 EM
Baselines: BERT-base Swin-Transformer CLIP ViLT CMGCN MSTI GPT-4 VideoChat Otter mPLUG-Owl
Builds on: Swin Transformer BERT ResNet YOLOX MSTI
Closest papers by meaning
- G^2SAM: Graph-Based Global Semantic Awareness Method for Multimodal Sarcasm Detection (AAAI 2024 · similarity 0.87)
- Well, Now We Know! Unveiling Sarcasm: Initiating and Exploring Multimodal Conversations with Reasoning (AAAI 2024 · similarity 0.87)
- Debiasing Multimodal Sarcasm Detection with Contrastive Learning (AAAI 2024 · similarity 0.86)
- Is Sarcasm Detection a Step-by-Step Reasoning Process in Large Language Models? (AAAI 2025 · similarity 0.85)
- Multi-Granular Multimodal Clue Fusion for Meme Understanding (AAAI 2025 · similarity 0.85)
- Debiased Multimodal Understanding for Human Language Sequences (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.