Research map / Vision-language and multimodal / Human and embodied modeling
Multi-Modal Discussion Transformer: Integrating Text, Images and Graph Transformers to Detect Hate Speech on Social Media
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes mDT, a graph-transformer model that interleaves BERT/ViT modality fusion layers via bottleneck tokens with discussion-graph layers using hierarchical spatial encodings, and introduces the HatefulDiscussions Reddit dataset for hate speech detection.
Problem
Hate speech detection on social media is typically done per comment with text only, ignoring images and the surrounding discussion context that can change a comment's meaning.
Approach
Interleave multi-modal fusion layers (BERT and ViT joined through shared bottleneck tokens) with graph transformer layers over the discussion tree, so each comment's multimodal representation is grounded in its discussion context; add a hierarchical spatial encoding based on Cantor pairing of upward and downward hop counts.
Key results
- mDT achieves 0.880 accuracy and 0.877 F1 on HatefulDiscussions, versus 0.858 F1 for BERT-HatefulDiscuss and 0.667 F1 for Graphormer.
- Removing images drops mDT accuracy from 0.880 to 0.832 and F1 from 0.877 to 0.828.
- Constraining graph attention to a 5-hop window gives the best result (0.880 accuracy), while unconstrained attention yields 0.870.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BERT-base, ViT-base and discussion graph context in place of the strongest text-only baseline, BERT-HatefulDiscuss | mDT · HatefulDiscussions | F1: 0.858 -> 0.877 | Helped |
| Add mDT's multimodal discussion-context modeling instead of Graphormer | HatefulDiscussions hate-speech detection · HatefulDiscussions | F1: 0.667 -> 0.877 | Helped |
| Use bottleneck size 8 instead of 4 (ablation) | mDT · HatefulDiscussions | F1: 0.877 -> 0.863 | Hurt |
| Use bottleneck size 16 instead of 4 (ablation) | mDT · HatefulDiscussions | F1: 0.877 -> 0.852 | Hurt |
| Use bottleneck size 32 instead of 4 (ablation) | mDT · HatefulDiscussions | F1: 0.877 -> 0.872 | Hurt |
| Constrain graph attention to 2 hops instead of 5 (ablation) | mDT · HatefulDiscussions | F1: 0.877 -> 0.866 | Hurt |
| Use unrestricted graph attention instead of a 5-hop window (ablation) | mDT · HatefulDiscussions | F1: 0.877 -> 0.855 | Hurt |
| Use 6 fusion layers instead of 12 (ablation) | mDT · HatefulDiscussions | F1: 0.877 -> 0.855 | Hurt |
| Use 8 fusion layers instead of 12 (ablation) | mDT · HatefulDiscussions | F1: 0.877 -> 0.855 | Hurt |
| Use 10 fusion layers instead of 12 (ablation) | mDT · HatefulDiscussions | F1: 0.877 -> 0.862 | Hurt |
| Remove images (ablation) | mDT · HatefulDiscussions | F1: 0.877 -> 0.828 | Hurt |
Surprising findings
- Adding discussion context can mislead the model when context comments are mostly neutral, causing errors where text-only BERT was correct.
- Performance gains from more fusion layers were non-linear: eight fusion layers outperformed ten.
Limitations
- Performance drops when discussion context is predominantly neutral, which can mislead the model.
- Only 424 labelled instances have an image attached; most images are in context rather than on the labelled comment.
- Comparisons with image-text multimodal methods were left to future work.
Experimental setup
Datasets: Contextual Abuse Dataset (CAD) Slurs corpus Learning to Intervene (LTI) dataset HatefulDiscussions
Benchmarks: HatefulDiscussions
Metrics: accuracy precision recall F1
Baselines: BERT-HateXplain Detoxify RoBERTa Dynabench BERT-HatefulDiscuss Graphormer
Builds on: Graphormer VilT bottleneck fusion transformer HateXplain
Closest papers by meaning
- TOT：Topology-Aware Optimal Transport for Multimodal Hate Detection (AAAI 2023 · similarity 0.88)
- An Information Retrieval Approach to Building Datasets for Hate Speech Detection (NeurIPS 2021 · similarity 0.88)
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (NeurIPS 2020 · similarity 0.88)
- TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection (AAAI 2026 · similarity 0.87)
- Spanning the Spectrum of Hatred Detection: A Persian Multi-Label Hate Speech Dataset with Annotator Rationales (AAAI 2024 · similarity 0.87)
- Hateful Meme Detection through Context-Sensitive Prompting and Fine-Grained Labeling (Student Abstract) (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.