Research map / Adversarial robustness and security / Black-box and transfer attacks
Confidence Elicitation: A New Attack Vector for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Confidence Elicitation Attacks (CEAttack), a hard-label black-box word-substitution attack on LLM classifiers that uses verbalized confidence elicited from the model as optimization feedback.
Problem
Black-box hard-label attacks on LLMs only see final predictions, so they need many queries and lack guidance during word-substitution optimization; soft-label attacks are considered unrealistic for closed-source APIs.
Approach
Elicit verbalized confidence from the LLM (two-step k-guess then verbal confidence prompt) and aggregate guesses and confidences into a Dirichlet-mean probability, then greedily keep word substitutions that lower the confidence in the true class, bottom-up until misclassification.
Key results
- CEAttack achieves ASR of 19.73% (SST2), 30.74% (AG-News), 45.67% (StrategyQA) on LLaMA-3-8B, outperforming SSPAttack (9.73, 26.71, 29.67).
- On Mistral-7B-Instruct-v0.3, CEAttack reaches ASR 17.94%, 38.33%, 39.26% on SST2, AG-News, StrategyQA, above SSPAttack.
- CEAttack uses far fewer successful-attack queries than SSPAttack (e.g., 25.60 vs 82.95 on SST2 with LLaMA-3-8B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use confidence elicitation to guide word-substitution attacks instead of SSPAttack's hard-label optimization | LLaMA-3-8B-Instruct · SST-2 | ASR: 9.73 -> 19.73 | Helped |
| Use confidence elicitation to guide word-substitution attacks instead of SSPAttack's hard-label optimization | LLaMA-3-8B-Instruct · AG-News | ASR: 26.71 -> 30.74 | Helped |
| Use confidence elicitation to guide word-substitution attacks instead of SSPAttack's hard-label optimization | LLaMA-3-8B-Instruct · StrategyQA | ASR: 29.67 -> 45.67 | Helped |
| Use confidence elicitation to guide word-substitution attacks instead of TextHoaxer's hard-label optimization | Mistral-7B-Instruct-v0.3 · SST-2 | ASR: 16.08 -> 17.94 | Helped |
| Use confidence elicitation to guide word-substitution attacks instead of TextHoaxer's hard-label optimization | Mistral-7B-Instruct-v0.3 · AG-News | ASR: 26.43 -> 38.33 | Helped |
| Use confidence elicitation to guide word-substitution attacks instead of SSPAttack's hard-label optimization | Mistral-7B-Instruct-v0.3 · StrategyQA | ASR: 30.99 -> 39.26 | Helped |
| Use CEAttack++ settings, including a delete-word ranking scheme and larger synonym search (ablation) | CEAttack on GPT-4o 2024-08-06 · SST-2 | ASR: 14.58 -> 28.42 | Helped |
| Use CEAttack++ settings, including a delete-word ranking scheme and larger synonym search (ablation) | CEAttack on GPT-4o 2024-08-06 · AG-News | ASR: 9.2 -> 14.77 | Helped |
| Use CEAttack++ settings, including a delete-word ranking scheme and larger synonym search (ablation) | CEAttack on GPT-4o 2024-08-06 · StrategyQA | ASR: 29.69 -> 36.76 | Helped |
| Increase the maximum number of perturbed words |W| and synonyms per word |S| (ablation) | CEAttack · SST-2, AG-News, and StrategyQA | ASR | Helped |
| Set temperature to τ=0.7 rather than approximately 0 (ablation) | CEAttack · Classification attack evaluation | ASR | Mixed |
Surprising findings
- Verbalized confidence elicitation from LLMs is reported as reasonably calibrated (ECE ~0.05-0.15 on core tasks), enabling confidence-guided attacks in pure black-box settings.
- Confidence elicitation may be at odds with adversarial robustness, since it exposes a usable optimization signal.
Limitations
- Results on StrategyQA show weak AUROC (~0.66) and calibration is inconsistent across models and datasets.
- Evaluation uses only word-substitution attacks on a small set of datasets, with a 500-sample evaluation subset.
- Temperature set near zero deviates from the default generation settings used in prior confidence elicitation work.
Experimental setup
Datasets: SST-2 AG-News StrategyQA RTE QNLI
Benchmarks: SST-2 AG-News StrategyQA
Metrics: clean accuracy accuracy under attack attack success rate semantic similarity perplexity queries attack time ECE AUROC
Baselines: Self-Fool Word Sub TextHoaxer SSPAttack
Builds on: SSPAttack TextHoaxer Self-Fool verbalized confidence elicitation counter-fitted embeddings
Closest papers by meaning
- Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling (NeurIPS 2026 · similarity 0.88)
- On the Robustness of Verbal Confidence of LLMs in Adversarial Attacks (NeurIPS 2025 · similarity 0.88)
- Scaling Trends in Language Model Robustness (ICML 2025 · similarity 0.87)
- Uncovering Safety Risks of Large Language Models through Concept Activation Vector (NeurIPS 2024 · similarity 0.87)
- REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective (ICML 2025 · similarity 0.86)
- Efficient Adversarial Training in LLMs with Continuous Attacks (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.