Research map / Adversarial robustness and security / Attacks on text and graphs
Scaling Trends in Language Model Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how adversarial robustness of Pythia and Qwen2.5 language-model classifiers scales with model size, attack compute, and adversarial-training compute, and projects the offense-defense balance.
Problem
Whether scaling model size and compute makes language models more or less robust to adversarial attacks, and whether attack or defense gains the advantage as both scale.
Approach
Applies a scaling lens to LM robustness: fits attack success rate (logit) against attack compute as a fraction of pretraining compute, and compares attack and adversarial-training compute scaling slopes to assess the offense-defense balance across model sizes.
Key results
- Larger Pythia models are more sample-efficient but less compute-efficient in adversarial training than smaller ones.
- Attack success rate improves smoothly with attack compute against both undefended and adversarially trained models; attack scaling outpaces adversarial training slopes (<1) across studied models.
- Larger models trained adversarially transfer robustness better to an infix threat model, though no model size transfers to a prefix attack.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model size without safety training | Finetuned Pythia and Qwen2.5 classifiers · Classification tasks under GCG, BEAST, and RandomToken attacks | attack success rate | Mixed |
| Increase model size | Qwen2.5 Instruct · StrongREJECT | attack success rate | Helped |
| Increase attack compute | Undefended and adversarially trained models · Classification tasks | attack success rate | Hurt |
| Adversarially train with GCG | Finetuned classifiers · Classification tasks | attack success rate: above 90% to below 20% after 5 rounds; below the 5% attack success rate threshold | Helped |
| Use larger models for adversarial training | Pythia and Qwen2.5 classifiers · Classification tasks | adversarial training rounds to reach the same robustness | Helped |
| Use larger models for adversarial training | Pythia and Qwen2.5 classifiers · Classification tasks | FLOPs to reach the same robustness | Hurt |
| Double attack compute while defense compute doubles | Attacker versus adversarially trained classifier · Spam | attack success rate | Hurt |
| Train against GCG and evaluate against BEAST | Adversarially trained classifiers · Spam and Harmless | attack success rate | Helped |
| Train against RandomToken and evaluate against GCG | Small classifiers · Classification tasks | attack success rate: above 95% to below 75% | Helped |
| Train against RandomToken and evaluate against GCG | Large classifiers · Classification tasks | attack success rate | No clear effect |
| Train against suffix GCG and evaluate against infix GCG | Larger Pythia classifiers · Spam | attack success rate | Helped |
| Train against suffix GCG and evaluate against prefix GCG | Pythia classifiers · Spam | attack success rate | No clear effect |
Surprising findings
- In the absence of safety training, larger models are not consistently more robust; on Spam, a 50x increase in parameters from 123.7M to 6.7B yields higher attack success rate.
- Small models benefit more than large models from adversarial training on a weak RandomToken attack when evaluated against GCG.
Limitations
- Focuses on classifiers rather than open-ended generative jailbreaks; generative results limited to Qwen2.5 Instruct up to 14B.
- Defense side limited to adversarial training; other defenses like latent-space methods or safeguard models not explored.
- Only two model families (Pythia, Qwen2.5) and sizes up to 14B due to compute constraints; Pythia FLOP estimation bug required manual adjustment.
Experimental setup
Datasets: Spam IMDB Helpful Harmless PasswordMatch WordLength StrongREJECT
Benchmarks: Spam IMDB Helpful Harmless PasswordMatch WordLength StrongREJECT
Metrics: attack success rate attack compute to reach 2% attack success rate logit attack success rate slope
Baselines: RandomToken BEAST
Builds on: GCG BEAST StrongREJECT Pythia
Closest papers by meaning
- Scaling Laws for Adversarial Attacks on Language Model Activations and Tokens (ICLR 2025 · similarity 0.92)
- Statistical Estimation of Adversarial Risk in Large Language Models under Best-of-N Sampling (NeurIPS 2026 · similarity 0.91)
- Efficient Adversarial Training in LLMs with Continuous Attacks (NeurIPS 2024 · similarity 0.88)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.88)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.88)
- REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.