Research map / Adversarial robustness and security / Attacks on text and graphs
LimeAttack: Local Explainable Method for Textual Hard-Label Adversarial Attack
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LimeAttack, a hard-label textual adversarial attack that ranks word importance with LIME over masked neighborhood samples and then runs beam search over synonym substitutions under a tiny query budget.
Problem
Textual adversarial attacks in the hard-label setting (only discrete predicted labels available) need many queries and their success is limited by random adversarial initialization, and word importance cannot be computed from confidence scores.
Approach
Use a local explainable method (LIME) on binary mask neighborhood samples, using only predicted labels, to estimate word importance, then perform beam search from the benign sample outward (inside-to-outside) with a sampling rule mixing high-similarity, low-similarity and random candidates.
Key results
- Under a query budget of 100, LimeAttack achieves ASR of 49.9% on CNN-MR versus 44.4% for HLBB and 49.4% for TextHacker, with lower or comparable perturbation rate.
- On BERT-SST-2 with budget 100, LimeAttack reaches 27.8% ASR vs 26.1% for LeapAttack.
- On ChatGPT with budget 100 (20 examples), LimeAttack ties for highest ASR at 20.0% with lower perturbation (4.51%) than LeapAttack and TextHacker.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LimeAttack instead of the strongest hard-label baseline | CNN, LSTM, and BERT text classifiers · MR, SST-2, AG, and Yahoo | ASR | Helped |
| use LimeAttack instead of the strongest hard-label baseline | BERT · SNLI, MNLIm, and MNLImm | ASR | Helped |
| add LIME word-importance ranking instead of randomly selecting words (ablation) | LimeAttack · MR (BERT) | ASR: 30.1 -> 39.3 | Helped |
| add LIME word-importance ranking instead of randomly selecting words (ablation) | LimeAttack · SST-2 (BERT) | ASR: 32.1 -> 36.5 | Helped |
| use mixed sampling rules instead of selecting highest-similarity candidates (ablation) | LimeAttack · MR (BERT) | ASR: 35.8 -> 39.3 | Helped |
| use mixed sampling rules instead of selecting lowest-similarity candidates (ablation) | LimeAttack · MR (BERT) | ASR: 31.5 -> 39.3 | Helped |
| use mixed sampling rules instead of random candidate selection (ablation) | LimeAttack · MR (BERT) | ASR: 32.1 -> 39.3 | Helped |
| replace LIME word ranking with SHAP (ablation) | LimeAttack · CNN-MR and BERT-SST-2, query budget 100 | ASR: 49.9 vs 49.7; 27.8 vs 27.7 | No clear effect |
| replace LIME word ranking with a decision tree (ablation) | LimeAttack · CNN-MR and BERT-SST-2, query budget 100 | ASR: 49.9 vs 50.1; 27.8 vs 27.9 | No clear effect |
| add adversarial training using LimeAttack examples (ablation) | CNN on MR · MR | ASR: 38.18 -> 35.09 | Hurt |
Surprising findings
- LIME-based word importance does not outperform deletion-based importance in score-based attacks (deletion gives lower perturbation rate), contrary to the expectation that local explanation would help.
- Greedily choosing the lowest-similarity candidate does not maximize final attack success rate.
Limitations
- Only tested on classification and entailment tasks; other NLP tasks such as dialogue or translation not studied.
- LLM evaluation limited to a few models and 100 MR samples due to API limits; LLaMA not tested.
- Requires more queries than some baselines when computing word importance on long texts, and lower semantic similarity than some baselines.
Experimental setup
Datasets: MR SST-2 AG Yahoo SNLI MNLI
Benchmarks: MR SST-2 AG Yahoo SNLI MNLI matched MNLI mismatched
Metrics: attack success rate perturbation rate semantic similarity query number grammatical errors
Baselines: HLBB TextHoaxer LeapAttack TextHacker TextFooler PWWS BERT-Attack
Builds on: LIME TextFooler HLBB TextHacker TextHoaxer LeapAttack
Closest papers by meaning
- Adversarial Training for Defense Against Label Poisoning Attacks (ICLR 2025 · similarity 0.85)
- An LLM can Fool Itself: A Prompt-Based Adversarial Attack (ICLR 2024 · similarity 0.85)
- Revisiting Character-level Adversarial Attacks for Language Models (ICML 2024 · similarity 0.85)
- Confidence Elicitation: A New Attack Vector for Large Language Models (ICLR 2025 · similarity 0.85)
- Adversarial Paraphrasing: A Universal Attack for Humanizing AI-Generated Text (NeurIPS 2025 · similarity 0.85)
- Semantic-Aware Multi-Label Adversarial Attacks (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.