Research map / Adversarial robustness and security / Attacks on text and graphs
On the Robustness of Verbal Confidence of LLMs in Adversarial Attacks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies robustness of LLM verbalized confidence under perturbation-based (VCA-TF, VCA-TB, Typos, SubSwapRemove) and jailbreak-based (ConfidenceTriggers, ConfidenceTriggers-AutoDAN) attacks, and tests existing defences.
Problem
Verbal confidence elicited from LLMs may be easily manipulated by adversarial inputs, causing confidence drops and answer changes; its robustness under attack is unstudied.
Approach
Adapts black-box text attacks (TextFooler, TextBugger) and a genetic-algorithm jailbreak (AutoDAN) so their objective is the verbalized confidence score rather than logit probabilities, and optimizes reusable confidence-reducing trigger tokens appended to the system prompt.
Key results
- VCA-TF/TB and SubSwapRemove reduce average confidence by up to ~30% on affected samples and flip answers in up to 100% of originally correct examples (SC-CA on SQA).
- ConfidenceTriggers achieves confidence drops up to 29.5% on Llama-3-8B, while ConfidenceTriggers-AutoDAN reaches up to 16.7% and random tokens baseline is weaker.
- Paraphrase and SmoothLLM defences change confidence on many benign prompts (differences up to ~43%) with minimal reduction in attack effectiveness, and perplexity filtering fails to separate adversarial from normal text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply query-targeted VCA-TF | QA prompts evaluated with Llama-3-8B and GPT-3.5 · TQA, MQA, and SQA | confidence reduction (Δ Cf.): 5.3 | Helped |
| Apply query-targeted SubSwapRemove | QA prompts evaluated with Llama-3-8B and GPT-3.5 · TQA, MQA, and SQA | confidence reduction (Δ Cf.): 6.9 | Helped |
| Apply query-targeted VCA-TB | QA prompts evaluated with Llama-3-8B and GPT-3.5 · TQA, MQA, and SQA | confidence reduction (Δ Cf.): 6.9 | Helped |
| Apply query-targeted Typos | QA prompts evaluated with Llama-3-8B and GPT-3.5 · TQA, MQA, and SQA | confidence reduction (Δ Cf.): 6.5 | Helped |
| Append optimized ConfidenceTriggers rather than random tokens | Llama-3-8B · Trigger evaluation | confidence reduction (Δ Cf.): 0.3 -> 10.3 | Helped |
| Append optimized ConfidenceTriggers rather than random tokens | GPT-3.5 · Trigger evaluation | confidence reduction (Δ Cf.): 0.8 -> 1.6 | Helped |
| Use ConfidenceTriggers-AutoDAN instead of random tokens | GPT-3.5 · Trigger evaluation | confidence reduction (Δ Cf.): 0.8 -> 6.7 | Helped |
| Use ConfidenceTriggers-AutoDAN instead of ConfidenceTriggers | Llama-3-8B · Trigger evaluation | confidence reduction (Δ Cf.): 10.3 -> 3.4 | Hurt |
| Use ConfidenceTriggers-AutoDAN instead of ConfidenceTriggers | GPT-3.5 · Trigger evaluation | confidence reduction (Δ Cf.): 1.6 -> 6.7 | Helped |
| Apply confidence-agnostic adversarial perturbations | Confidence elicitation methods on SST-2 · SST-2 | confidence reduction: at most a 5% drop | Hurt |
| Use paraphrasing as a defense against ConfidenceTriggers-AutoDAN | Llama-3-8B · SQA, TQA, and MQA | confidence reduction on attacked prompts (Δ Adv.): 28.7; 23.2; 44.0 | No clear effect |
| Apply SmoothLLM as a defense against ConfidenceTriggers-AutoDAN | Llama-3-8B · SQA, TQA, and MQA | confidence reduction on attacked prompts (Δ Adv.): 15.8; 5.8; 3.6 | No clear effect |
Surprising findings
- Confidence-agnostic attacks (AdvGLUE) change confidence by at most ~5%, whereas confidence-targeted attacks are far more effective.
- Commonly used defences are largely ineffective or counterproductive, often altering confidence on benign inputs.
Limitations
- Only character and word-level attacks studied; sentence-level attacks left for future work.
- Larger/frontier models tested only on a subset due to attack cost.
- Datasets have low ecological validity.
Experimental setup
Datasets: MedMCQA TruthfulQA StrategyQA SST-2 AdvGLUE Sentiment140
Benchmarks: MedMCQA TruthfulQA StrategyQA SST-2 (AdvGLUE)
Metrics: average confidence % affected samples average confidence reduction on affected samples % answer flips on originally correct % answer flips on originally wrong Brier score AUROC
Baselines: Random tokens baseline Paraphrase defence SmoothLLM Perplexity filter LLM-Guard
Builds on: TextFooler TextBugger AutoDAN Self-consistency prompting SmoothLLM
Closest papers by meaning
- An LLM can Fool Itself: A Prompt-Based Adversarial Attack (ICLR 2024 · similarity 0.90)
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.88)
- Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts? (ICLR 2025 · similarity 0.88)
- Learning to Route LLMs with Confidence Tokens (ICML 2025 · similarity 0.88)
- AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs (ICML 2025 · similarity 0.88)
- Confidence Elicitation: A New Attack Vector for Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.