Research map / Adversarial robustness and security / Certified and empirical robustness
Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Off
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes perturbation-invariant adversarial training (PIAT), a regularized surrogate loss with a supervised pairwise natural loss plus a semi-supervised perturbation-invariance loss, to improve the effectiveness-robustness trade-off of neural ranking models against word substitution ranking attacks.
Problem
Adversarial training for neural ranking models against word substitution ranking attacks reduces effectiveness on clean data, creating a trade-off between effectiveness and adversarial robustness that is not well characterized for ranking.
Approach
Decomposes robust ranking error into natural and boundary ranking errors, shows perturbation invariance upper-bounds the boundary error, and trains with a regularized surrogate loss combining a supervised pairwise loss with a semi-supervised rank-invariance loss (KL, ListNet, or ListMLE variants).
Key results
- On BERT, PIAT_ListNet achieves RobustMRR@10 of 0.3728 and ASR of 36.1%, versus 0.3451 and 55.1% for vanilla adversarial training (AT).
- On ConvKNRM, PIAT_ListMLE reaches CleanMRR@10 of 0.2534 versus 0.2316 for AT, while improving robustness over AT.
- On PROP, PIAT_ListNet lowers ASR to 35.0% and LSD to 6.2 compared with 51.2% and 12.8 for AT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply PIAT ListNet instead of adversarial training | ConvKNRM · MS MARCO Passage Ranking | RobustMRR@10: 0.1896 -> 0.2035 | Helped |
| Apply PIAT ListNet instead of adversarial training | BERT · MS MARCO Passage Ranking | CleanMRR@10: 0.3743 -> 0.3892 | Helped |
| Apply PIAT ListNet instead of adversarial training | BERT · MS MARCO Passage Ranking | ASR: 55.1 -> 36.1 | Helped |
| Apply PIAT ListNet instead of adversarial training | PROP · MS MARCO Passage Ranking | RobustMRR@10: 0.3532 -> 0.3794 | Helped |
| Use KL divergence instead of ListNet for the adversarial ranking loss (ablation) | PIAT on ConvKNRM · MS MARCO Passage Ranking | RobustMRR@10: 0.2035 -> 0.2008 | Hurt |
| Use ListMLE instead of ListNet for the adversarial ranking loss (ablation) | PIAT on ConvKNRM · MS MARCO Passage Ranking | RobustMRR@10: 0.2035 -> 0.2018 | Hurt |
| Use KL divergence instead of ListNet for the adversarial ranking loss (ablation) | PIAT on BERT · MS MARCO Passage Ranking | RobustMRR@10: 0.3728 -> 0.3686 | Hurt |
| Use ListMLE instead of ListNet for the adversarial ranking loss (ablation) | PIAT on BERT · MS MARCO Passage Ranking | RobustMRR@10: 0.3728 -> 0.3705 | Hurt |
| Use KL divergence instead of ListNet for the adversarial ranking loss (ablation) | PIAT on PROP · MS MARCO Passage Ranking | RobustMRR@10: 0.3794 -> 0.3749 | Hurt |
| Use ListMLE instead of ListNet for the adversarial ranking loss (ablation) | PIAT on PROP · MS MARCO Passage Ranking | RobustMRR@10: 0.3794 -> 0.3767 | Hurt |
Surprising findings
- Standard training on BM25-level comparisons shows CertDR has lower clean effectiveness than undefended models, yet the paper reports it is more consistent between clean and adversarial rankings.
Limitations
- Only considers the word substitution ranking attack (WSRA).
- Constructing adversarial training examples is time-consuming, so adversarial training used only 1/10 of training queries.
Experimental setup
Datasets: MS MARCO passage ranking
Benchmarks: MS MARCO passage ranking
Metrics: CleanMRR@10 CleanMRR@100 RobustMRR@10 RobustMRR@100 ASR LSD
Baselines: standard training data augmentation vanilla adversarial training CertDR
Builds on: adversarial training ListNet ListMLE word substitution ranking attack
Closest papers by meaning
- Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data (ICLR 2025 · similarity 0.88)
- Maximization of Average Precision for Deep Learning with Adversarial Ranking Robustness (NeurIPS 2023 · similarity 0.88)
- Attribute-Guided Adversarial Training for Robustness to Natural Perturbations (AAAI 2021 · similarity 0.87)
- How Many Perturbations Break This Model? Evaluating Robustness Beyond Adversarial Accuracy (ICML 2023 · similarity 0.87)
- GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models (NeurIPS 2024 · similarity 0.87)
- Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.