Research map / Adversarial robustness and security / Backdoor and poisoning

Meme Trojan: Backdoor Attacks Against Hateful Meme Detection via Cross-Modal Triggers

AAAI 2025 · Adversarial robustness and security › Backdoor and poisoning · approach: Trigger-based backdoor attacks · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Meme Trojan, a backdoor attack on hateful meme detectors using a text-like cross-modal trigger ("..") placed adaptively via OCR and refined by a learnable trigger augmentor, which attacks both image and text encoders.

Problem

Hateful meme detectors are vulnerable to backdoor attacks during training on third-party data; existing uni-modal or dual-key (TrojVQA) backdoors fail in automatic OCR-based detection pipelines and reduce stealthiness.

Approach

A cross-modal trigger shaped like text ("..") embedded in the image is recognized by OCR and thus activates both the visual and textual branches; its position and size adapt to the meme's existing text, and a learned trigger augmentor (blended with the initial dots) reduces false activation on real punctuation.

Key results

• Under automatic OCR detection on FBHM, CMT reaches ASR 0.967 with Late Fusion and 0.856 with MMFT, versus TrojVQA's 0.678 and 0.811 for the same models (TrojVQA's ASR is higher for some models on other datasets).
• Against Neural Polarizer defense on FBHM with VisualBert, CMT keeps ASR 1.000 while TrojVQA drops to 0.167.
• CMT achieves image-level SSIM 0.9990 and LPIPS 0.0031, versus TrojVQA's SSIM 0.9923 and LPIPS 0.0198.

Measured effects reported in the paper

Surprising findings

• Under automatic OCR-based detection, most baselines (including TrojVQA) reach much lower ASR than under manual text entry, indicating OCR errors substantially hinder text-trigger activation.
• The CMT w/o TA variant achieves the highest ASR in some automatic-OCR settings because the text extractor recognizes its fixed form easily, while CMT gives higher CDA.

Limitations

• Performance depends on OCR accuracy; logos, skewed text and handwriting cause failed trigger placement and lower ASR.
• Attack effectiveness is lower on MAMI, where complex text makes text-like triggers hard to activate.
• Evaluated only with the attacker targeting non-hateful label on three datasets and six detectors; defenses were only discussed, not evaluated in depth.

Experimental setup

Datasets: FBHM MAMI HarMeme

Benchmarks: FBHM MAMI HarMeme

Metrics: CDA ASR PSNR SSIM LPIPS delta PPL delta GE USE

Baselines: TrojVQA CMT w/o TA FIBA Consider-like text trigger Red pattern trigger Random pattern trigger

Builds on: TrojVQA BadNet BadNL FIBA Neural Polarizer

Closest papers by meaning

• Hateful Meme Detection through Context-Sensitive Prompting and Fine-Grained Labeling (Student Abstract) (AAAI 2025 · similarity 0.83)
• Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Models (NeurIPS 2026 · similarity 0.82)
• TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detection (AAAI 2026 · similarity 0.82)
• Backdoor Attack on Propagation-based Rumor Detectors (AAAI 2025 · similarity 0.81)
• CL-Attack: Textual Backdoor Attacks via Cross-Lingual Triggers (AAAI 2025 · similarity 0.80)
• Silent Branding Attack: Trigger-free Data Poisoning Attack on Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.80)