Research map / Adversarial robustness and security / Attacks on text and graphs
Scaling Laws for Adversarial Attacks on Language Model Activations and Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that adversarially perturbing a few token activations in a language model can force exact prediction of up to ~1000 subsequent tokens, with a linear scaling law t_max = κ·a across Pythia, Phi and TinyStories models.
Problem
Understanding how susceptible language models are to adversarial attacks on their continuous activations (residual stream), and how control over output tokens scales with attack size and model dimension, in contrast to token-level attacks.
Approach
Treats the LLM as a controllable classifier from input activations to t-token continuations, and proposes an empirical scaling law t_max = κ f a / n with attack multiplier κ scaling linearly with activation dimension d, yielding a roughly constant attack resistance χ (~16-25) across model sizes. Compares this with greedy token-substitution attacks and shows the dimensionality-ratio prediction roug
Key results
- Attack multiplier κ = 119.2 ± 2.9 for Pythia-1.4B: one token of activations controls ≈119 output tokens.
- Attack resistance χ stays roughly constant at ≈16–25 bits across Pythia models from 70M to 2.8B (κ/d between 16.0 and 24.4).
- Token-substitution attacks need ≈8 input tokens to control one output token on Pythia-70M, versus ≈0.04 tokens for activation attacks; κ_token/κ ≈ 5e-3 vs predicted 2e-3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of controlled input activations | Activation attacks · Random target-token prediction across tested language models | maximum target length at 50% attack success: t_max = κ a | Helped |
| Use a larger activation dimension | Activation attacks · Pythia model suite | attack multiplier κ: κ / d ... between 16.0 ± 0.5 and 24.4 ± 0.5 | Helped |
| Replace activation perturbations with greedy token substitutions | Pythia-70M attacks · Random target-token prediction | attack multiplier κ: 24.2 ± 0.8 vs 0.12 | Hurt |
| Use the same perturbation across more context-target pairs (ablation) | Activation attacks · Pythia-1.4B, random target-token prediction | maximum target length at 50% attack success: t_max = κ f a / n | Hurt |
| Use only a fraction of activation dimensions (ablation) | Activation attacks · Pythia-1.4B, random target-token prediction | maximum target length at 50% attack success: t_max = κ f a / n | Hurt |
| Vary attack length, multiplicity, and dimension fraction at the same effective attack strength (ablation) | Activation attacks · Pythia-1.4B, random target-token prediction | attack multiplier κ: 128.4 ± 3.7; 119.2 ± 2.9 | No clear effect |
| Increase the context separating the attack from the target beyond 100 tokens (ablation) | Activation attacks · Pythia-70M, random target-token prediction | attack multiplier κ: κ ≈ 8 | Hurt |
Surprising findings
- Attack strength is essentially unaffected by up to 100 tokens of separation between attack and target, and still gives κ ≈ 8 at 2000 tokens of separation.
- Attacks on the activations are much stronger than token attacks, yet their attack resistance χ is within a factor ~2.6 of token attacks once dimensionality is accounted for.
Limitations
- Optimization uses unconstrained perturbations with no norm bound, so results are not directly comparable to norm-bounded image attacks.
- Experiments use random token contexts and targets rather than natural language, and only a few model families (Pythia, Phi, TinyStories).
- Attacks are applied only at the embedding-layer activations in main experiments; the threat model (retrieval, multimodal inputs) is argued rather than tested.
Experimental setup
Metrics: attack success rate p(a,t) t_max at 50% success attack multiplier κ attack resistance χ
Baselines: greedy token substitution attack
Builds on: Adam multi-attack adversarial examples activation addition
Closest papers by meaning
- Scaling Trends in Language Model Robustness (ICML 2025 · similarity 0.92)
- Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling (NeurIPS 2026 · similarity 0.89)
- Many-shot Jailbreaking (NeurIPS 2024 · similarity 0.88)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.87)
- Bits Leaked per Query: Information-Theoretic Bounds for Adversarial Attacks on LLMs (NeurIPS 2025 · similarity 0.87)
- Semantic Representation Attack against Aligned Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.