Research map / Image generation / Text-to-image diffusion

Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Models

CVPR 2025 · Image generation › Text-to-image diffusion · approach: Text generation and structured prediction · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that typographic text inserted into input images can steer CLIP-guided diffusion image-to-image generation toward harmful, biased, or neutral content, that existing text-focused defenses miss it, and releases the VMT-IGMs benchmark dataset.

Problem

Safety defenses for image generation models focus on the language modality, leaving the vision modality open to typographic attacks in image-to-image tasks that edit real-world images.

Approach

Applies typographic attacks (text rendered into input images) to CLIP-guided diffusion models in I2I tasks, varying typo factors and content categories, and shows current prompt and post-generation defenses fail to catch them; packages this as the VMT-IGMs benchmark built on CelebA-HQ.

Key results

• Typographic text raises CLIP Score alignment with inserted words by about 5.2 on average across UnCLIP, SD1.5, SDXL and FLUX (e.g., nouns average gain 5.28 vs adjectives 1.19).
• Visible typos with harmful words ('bloody') raise CLIP score by 3.04 on average, and neutral 'hat' by 5.32; invisible typos in black borders still shift scores (e.g., 'African' +1.18).
• The safety checker detects only 16.6% of harmful-typo images on average and 4.0% for bias typos, versus 23.7% for naked on UnCLIP-level breakdowns.

Measured effects reported in the paper

Surprising findings

• Prompt-level 'ignore text' instruction does not reduce the typographic influence on generated images.
• Pre-generation text prompt detectors all fail on benign prompts when the threatening content is carried by typography in the input image.

Limitations

• Evaluation limited to 1000 CelebA-HQ face images and a small set of prompts and typo words.
• Relies on CLIP Score as the main metric, which measures alignment with typo text rather than harmfulness of outputs.
• Defense evaluation covers a limited set of defenses and does not propose a new defense.

Experimental setup

Datasets: CelebA-HQ

Benchmarks: VMT-IGMs

Metrics: CLIP Score FID defense rate of safety checker

Baselines: Text Blacklist Detoxify CLIPScore prompt check LLM-based prompt check Latent Guard Safety checker Prompt-modified defense (ignore text)

Builds on: UnCLIP IP-Adapter typographic attack

Closest papers by meaning

• Attention! Your Vision Language Model Could Be Maliciously Manipulated (NeurIPS 2025 · similarity 0.87)
• Jailbreaking Vision-Language Models Through the Visual Modality (ICML 2026 · similarity 0.86)
• Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models (CVPR 2025 · similarity 0.86)
• Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.86)
• Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks (CVPR 2025 · similarity 0.86)
• When Understanding Becomes a Risk: Authenticity and Safety Risks in the Emerging Image Generation Paradigm (CVPR 2026 · similarity 0.86)