Research map / Vision-language and multimodal / Human and embodied modeling
Hateful Meme Detection through Context-Sensitive Prompting and Fine-Grained Labeling (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an end-to-end conceptual optimization framework (modalities, prompting, labeling, fine-tuning) for hateful meme detection, evaluated via a 12-configuration ablation of InternVL 8B and DistilBERT on the Hateful Memes dataset.
Problem
Hateful meme detection on social media is hard for multimodal classifiers, and it is unclear whether fine-grained prompting and labeling outweigh the effect of fine-tuning in vision-language models.
Approach
An end-to-end conceptual framework that treats performance as a multivariate function of modalities, prompting, labeling, and fine-tuning, and ablates these factors jointly (2x2 prompt-by-label matrix plus fine-tuning and modality) to show isolated optimizations are not sufficient on their own.
Key results
- Best configuration (fine-tuned multimodal InternVL 8B with category prompt and binary labels) reaches 68.933% accuracy and 66.827% AUROC on Hateful Memes, +19.611 and +19.449 points over the baseline.
- Ablations show fine-tuning, categorical prompting, and binary labels jointly drive the gain; scaled outputs help in most settings but not the best model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune InternVL 8B with a category prompt and binary labels instead of prompting it without fine-tuning | InternVL 8B · Hateful Memes evaluation | accuracy: 62.500 -> 68.933 | Helped |
| Fine-tune InternVL 8B with a category prompt and binary labels instead of prompting it without fine-tuning | InternVL 8B · Hateful Memes evaluation | AUROC: 62.199 -> 66.827 | Helped |
| Use a category prompt instead of a simple prompt (ablation) | Fine-tuned InternVL 8B with binary labels · Hateful Memes evaluation | accuracy: 68.233 -> 68.933 | Helped |
| Use binary labels instead of scale labels (ablation) | Fine-tuned InternVL 8B with a category prompt · Hateful Memes evaluation | accuracy: 65.933 -> 68.933 | Helped |
| Use binary labels instead of scale labels (ablation) | Fine-tuned InternVL 8B with a category prompt · Hateful Memes evaluation | AUROC: 63.139 -> 66.827 | Helped |
| Use scale labels instead of binary labels (ablation) | Prompted InternVL 8B with a simple prompt · Hateful Memes evaluation | accuracy: 61.500 -> 62.100 | Helped |
| Use a category prompt instead of a simple prompt (ablation) | Fine-tuned DistilBERT 66M with binary labels · Hateful Memes evaluation | accuracy: 61.033 -> 49.300 | Hurt |
| Use a category prompt instead of a simple prompt (ablation) | Fine-tuned DistilBERT 66M with binary labels · Hateful Memes evaluation | AUROC: 57.783 -> 47.378 | Hurt |
| Use scale labels instead of binary labels (ablation) | Fine-tuned InternVL 8B with a simple prompt · Hateful Memes evaluation | accuracy: 68.233 -> 65.367 | Hurt |
| Use scale labels instead of binary labels (ablation) | Fine-tuned DistilBERT 66M with a simple prompt · Hateful Memes evaluation | accuracy: 61.033 -> 60.833 | No clear effect |
| Use a category prompt instead of a simple prompt (ablation) | Prompted InternVL 8B with binary labels · Hateful Memes evaluation | accuracy: 61.500 -> 62.500 | Helped |
| Use scale labels instead of binary labels (ablation) | Prompted InternVL 8B with a category prompt · Hateful Memes evaluation | accuracy: 62.500 -> 62.700 | No clear effect |
Surprising findings
- Scaled (interval) labels generally improve performance but not for the best configuration, so isolated optimizations do not always transfer.
- The highest-complexity configuration was not the best-performing model.
Limitations
- Single dataset (Hateful Memes) and a single VLM family evaluated.
- Some configurations (e.g., unimodal category-prompt DistilBERT) were not tuned to convergence, with the loss decreasing slowly under fixed hyperparameters.
- Framework is conceptual; gains are not isolated from hyperparameter and configuration variance.
Experimental setup
Datasets: Hateful Memes
Benchmarks: Hateful Memes
Metrics: accuracy precision recall F1-score AUROC
Baselines: DistilBERT (unimodal) InternVL 8B prompting-only
Builds on: multi-stage prompting InternVL 2 DistilBERT
Closest papers by meaning
- TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection (AAAI 2026 · similarity 0.89)
- Multi-Modal Discussion Transformer: Integrating Text, Images and Graph Transformers to Detect Hate Speech on Social Media (AAAI 2024 · similarity 0.87)
- Multi-Granular Multimodal Clue Fusion for Meme Understanding (AAAI 2025 · similarity 0.85)
- ICM-Assistant: Instruction-tuning Multimodal Large Language Models for Rule-based Explainable Image Content Moderation (AAAI 2025 · similarity 0.85)
- Spanning the Spectrum of Hatred Detection: A Persian Multi-Label Hate Speech Dataset with Annotator Rationales (AAAI 2024 · similarity 0.84)
- Meme Trojan: Backdoor Attacks Against Hateful Meme Detection via Cross-Modal Triggers (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.