Research map / Vision-language and multimodal / Multimodal LLMs

TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TRACE, a hierarchical framework that augments hateful meme text with visually grounded LVLM captions, selects hate-relevant captions with a Gumbel-softmax caption scorer, and fine-tunes the last 4 CLIP text-encoder layers, reaching SoTA-level accuracy on FHM.

Problem

Hateful meme detection is hard because image and overlaid text combine into culturally nuanced meanings and benign confounders mislead unimodal or simple fused models.

Approach

Visually grounded (RAM tags + GroundingDINO boxes) LVLM caption augmentation, a caption-scoring network with Gumbel-softmax selection and an auxiliary hate-relevance loss, combined with partial fine-tuning of deeper CLIP text-encoder layers instead of projection layers.

Key results

• On FHM, CLIP-XLM-R-ViT-H/14 with last-4 text-encoder layers tuned and L_cls+L_rel achieves accuracy 0.807 and F1 0.806, comparable to PALI-X-VPD (Acc 0.808).
• Projection-layer-only fine-tuning of CLIP-ViT-L/14 reaches only 0.720 accuracy, below Hate-CLIPper (0.756) and RGCL (0.788).
• On MultiOFF, TRACE reaches F1 0.673 versus 0.540 for the original baseline.

Surprising findings

• Simple projection-layer fine-tuning on generated captions was insufficient, while tuning deeper text-encoder layers gave large gains.
• Zero-shot LLM classification of LVLM-generated captions was significantly inferior to direct LVLM zero-shot classification.

Limitations

• Sensitive to small or occluded objects (e.g., guns) missed by visual grounding, leading to false negatives.
• Still tunes millions of parameters and depends on multiple large models and APIs; efficiency gains are relative to PALI-X-VPD.
• Evaluation on small test sets (MultiOFF 149 test samples) and single FHM test split.

Experimental setup

Datasets: Hateful Memes (FHM) MultiOFF

Benchmarks: Hateful Memes (FHM) MultiOFF

Metrics: accuracy F1 precision recall

Baselines: Hate-CLIPper RGCL PALI-X-VPD HateSieve MultiOFF baseline CLIP-ViT-L/14 projection fine-tuning LLM2CLIP-Llama-3.2

Builds on: CLIP Hate-CLIPper RGCL RAM GroundingDINO

Closest papers by meaning

• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes (NeurIPS 2020 · similarity 0.90)
• Hateful Meme Detection through Context-Sensitive Prompting and Fine-Grained Labeling (Student Abstract) (AAAI 2025 · similarity 0.89)
• Multi-Modal Discussion Transformer: Integrating Text, Images and Graph Transformers to Detect Hate Speech on Social Media (AAAI 2024 · similarity 0.87)
• Spanning the Spectrum of Hatred Detection: A Persian Multi-Label Hate Speech Dataset with Annotator Rationales (AAAI 2024 · similarity 0.86)
• A Multi-Task Benchmark for Abusive Language Detection in Low-Resource Settings (NeurIPS 2025 · similarity 0.85)
• Multi-Granular Multimodal Clue Fusion for Meme Understanding (AAAI 2025 · similarity 0.84)