Research map / Large language models / Language, safety and interpretability

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection

ICML 2025 · Large language models › Language, safety and interpretability · approach: Jailbreak elicitation and risk · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that Judge LLMs are vulnerable to token segmentation bias and proposes Emoji Attack, which uses in-context learning to make the target LLM insert emojis into harmful responses so Judge LLMs classify them as safe.

Problem

Judge LLMs used to flag harmful responses from target LLMs can be fooled by small input perturbations that change tokenization, letting jailbreak outputs evade detection.

Approach

Identifies token segmentation bias (delimiters split words into sub-tokens that distort embeddings) and extends it to emojis, which add sub-token changes plus semantic ambiguity; a black-box version induces emoji insertion via in-context learning in the target LLM's outputs, and a white-box version picks insertion positions minimizing embedding cosine similarity.

Key results

• With Emoji Attack on CodeChameleon prompts, the weighted-average unsafe prediction ratio across Judge LLMs drops from 62.4% to 50.4%.
• On the 1,432 harmful-response set, white-box optimized-position emoji insertion lowers the average unsafe prediction ratio from 84.4% (default) to 65.2%, with ShieldLM falling to 3.0%.
• Deepinception with Emoji Attack drops ShieldLM's unsafe prediction ratio from 71.9% to 3.5%.

Measured effects reported in the paper

Surprising findings

• Emoji insertion with a toxic emoji (e.g., middle finger) can increase the chance of unsafe predictions in commercial judges, showing semantic meaning of emojis cuts both ways.
• GPT-4 remains largely robust to emoji insertion, and DeepSeek shows little reduction, while the Emoji Attack sometimes increases unsafe ratios (e.g., Llama Guard 2 on Deepinception).

Limitations

• One-shot in-context example sometimes yields insufficient emoji insertion, leading to increased unsafe ratios in some cases.
• White-box position selection uses a surrogate model (gtr-t5-xl) rather than the judge's own embeddings, and the attack is evaluated mostly on a single target LLM (gpt-3.5-turbo).
• Some commercial judges (GPT-4, DeepSeek) are largely robust to the attack.

Experimental setup

Datasets: AdvBench LLM Self Defense Red Teaming Attempts EasyJailbreak jailbreak prompts 402 offensive phrases list

Benchmarks: EasyJailbreak jailbreak prompts (Deepinception, ReNellm, Jailbroken, CodeChameleon) AdvBench and jailbreak-generated harmful responses (1,432 total)

Metrics: unsafe prediction ratio

Baselines: Deepinception ReNellm Jailbroken CodeChameleon GCG

Builds on: GCG EasyJailbreak in-context learning

Closest papers by meaning

• Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models (AAAI 2025 · similarity 0.89)
• An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks (ICML 2025 · similarity 0.87)
• Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.87)
• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (NeurIPS 2024 · similarity 0.87)
• SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism (NeurIPS 2025 · similarity 0.86)
• Efficient LLM Jailbreak via Adaptive Dense-to-sparse Constrained Optimization (NeurIPS 2024 · similarity 0.86)