Research map / Large language models / Language, safety and interpretability

FlipAttack: Jailbreak LLMs via Flipping

ICML 2025 · Large language models › Language, safety and interpretability · approach: Jailbreak attacks and defenses · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FlipAttack, a single-query black-box jailbreak that disguises harmful prompts by flipping word order or characters and uses a guidance prompt (CoT, role-play, few-shot) to have the LLM recover and execute the request.

Problem

Existing jailbreak attacks on black-box LLMs need white-box gradients, many iterative queries, or complex auxiliary tasks (ciphers, code) that LLMs struggle to execute, and they are often detected by guard models.

Approach

Exploits the left-to-right autoregressive understanding of LLMs: a left-side perturbation built only from the prompt's own characters (flipping words or characters) disrupts comprehension and evades guard models, while a flipping guidance module teaches the victim LLM to reverse the flip and execute the request in one query.

Key results

• FlipAttack achieves 78.97% average ASR-GPT across 8 LLMs, versus 56.64% for the runner-up ReNeLLM.
• FlipAttack reaches 94.04% ASR-GPT on GPT-4 Turbo and 88.08% on Claude 3.5 Sonnet.
• FlipAttack bypasses guard models with a 98.08% average bypass rate across 5 guard models.

Measured effects reported in the paper

Surprising findings

• Weaker models such as GPT-3.5 Turbo and LLaMA 3.1 405B struggle with the flipping task without few-shot demonstrations, while CoT can reduce ASR on GPT-4o mini (near 0).
• Adding left-side perturbation raised perplexity much more than right-side perturbation across open LLMs and guard models.

Limitations

• Performance on LLaMA 3.1 405B (27.50% ASR-GPT) and GPT-4o mini (61.92%) is substantially lower than on strong models.
• Few-shot demonstrations may reveal harmful words in split form, increasing detection risk; a better splitting method is left open.
• Defenses tested (system prompt defense and perplexity-based filtering) were found ineffective, and the paper notes that the comparison with baselines using system prompts may not be fair.

Experimental setup

Datasets: AdvBench harmful behaviors Alpaca safe subset

Benchmarks: AdvBench harmful behaviors StrongREJECT

Metrics: attack success rate (ASR-GPT) attack success rate (ASR-DICT) bypass rate perplexity token count match rate

Baselines: GCG AutoDAN MAC COLD-Attack PAIR TAP Base64 GPTFuzzer DeepInception DRA ArtPrompt PromptAttack

Builds on: GCG AutoDAN ReNeLLM CodeChameleon JailbreakBench

Closest papers by meaning

• JailPO: A Novel Black-Box Jailbreak Framework via Preference Optimization Against Aligned LLMs (AAAI 2025 · similarity 0.89)
• One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs (ICLR 2025 · similarity 0.88)
• Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.87)
• Tree of Attacks: Jailbreaking Black-Box LLMs Automatically (NeurIPS 2024 · similarity 0.87)
• Injecting Universal Jailbreak Backdoors into LLMs in Minutes (ICLR 2025 · similarity 0.87)
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (ICLR 2024 · similarity 0.87)