Research map / Vision-language and multimodal / Human and embodied modeling
Well, Now We Know! Unveiling Sarcasm: Initiating and Exploring Multimodal Conversations with Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the SIRC task (locating sarcasm initiation spans and generating reasoning for sarcastic multimodal conversations in English and Hinglish), the SIRD dataset, and SIRG, a BART-based multimodal model with a cross-attention shared-fusion module.
Problem
Sarcasm detection alone does not reveal where sarcasm starts or why an utterance is sarcastic; existing work lacks multimodal conversational data and methods for sarcasm initiation and reasoning in English and code-mixed dialogue.
Approach
Jointly trains sarcasm detection, initiation span extraction, and reasoning generation on a BART backbone, fusing text, audio (OpenSMILE) and video (3D-ResNeXt) features via a cross-attention "Shared-Fusion" adapter inserted into BART layers.
Key results
- SIRG-TAVB reaches 52.22 ROUGE-L and 25.13 BLEU-4 on SIRD-English reasoning, versus 45.26 and 17.23 for MAF-TAVB.
- On SIRD-English sarcasm detection, SIRG-TAVB obtains F1 84.11 versus 77.21 for CMSEKI.
- Human evaluation on 250 samples rates SIRG-M informativeness at 3.83 versus 2.97 for MAF-TAVB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SIRG-TAVB instead of MAF-TAVB | sarcasm reasoning generation · SIRD-English | ROUGE-L: 45.26 -> 52.22 | Helped |
| use SIRG-TAVB instead of MAF-TAVB | sarcasm reasoning generation · SIRD-Hinglish | BLEU-4: 8.57 -> 16.32 | Helped |
| use SIRGB instead of CMSEKI | sarcasm detection · SIRD-English | F1: 77.21 -> 84.11 | Helped |
| use SIRGB instead of CMSEKI | sarcasm detection · SIRD-Hinglish | F1: 64.52 -> 69.99 | Helped |
| add Shared-Fusion rather than remove it (ablation) | SIRG · SIRD-English | BERTScore: 83.24 -> 86.11 | Helped |
| replace the clustering variant with the full method (ablation) | SIRG · SIRD-English | BERTScore: 83.98 -> 86.11 | Helped |
| use the full modality configuration rather than the T-A variant (ablation) | SIRG · SIRD-English | BERTScore: 83.13 -> 86.11 | Helped |
| use the full modality configuration rather than the T-V variant (ablation) | SIRG · SIRD-English | BERTScore: 82.45 -> 86.11 | Helped |
Limitations
- Small dataset (690 English and 2255 Hinglish samples) derived from TV sitcoms
- Reasoning and span annotations rely on two or three annotators with moderate agreement (82.48% average cosine similarity)
Experimental setup
Datasets: MUStARD MASAC SIRD
Benchmarks: SIRD-English SIRD-Hinglish
Metrics: FM PM HD JF ROS F1 Accuracy ROUGE-L BLEU-3 BLEU-4
Baselines: BiRNN-Attention CNN-GRU BiRNN-HateXplain BERT-HateXplain SpanBERT CMSEKI Pointer Generator Network BART mBART MAF-TAVM MAF-TAVB
Builds on: BART MAF CMSEKI MUStARD
Closest papers by meaning
- Is Sarcasm Detection a Step-by-Step Reasoning Process in Large Language Models? (AAAI 2025 · similarity 0.91)
- DocMSU: A Comprehensive Benchmark for Document-Level Multimodal Sarcasm Understanding (AAAI 2024 · similarity 0.87)
- SEVADE: Self-Evolving Multi-Agent Analysis with Decoupled Evaluation for Hallucination-Resistant Sarcasm Detection (AAAI 2026 · similarity 0.86)
- Debiasing Multimodal Sarcasm Detection with Contrastive Learning (AAAI 2024 · similarity 0.86)
- G^2SAM: Graph-Based Global Semantic Awareness Method for Multimodal Sarcasm Detection (AAAI 2024 · similarity 0.85)
- MIntRec2.0: A Large-scale Benchmark Dataset for Multimodal Intent Recognition and Out-of-scope Detection in Conversations (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.