Research map / Adversarial robustness and security / Attacks on text and graphs
Like Oil and Water: Group Robustness Methods and Poisoning Defenses May Be at Odds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that group robustness methods (JTT, GEORGE) amplify poisons while poisoning defenses (EPIc, federated robust aggregation) eliminate minority samples, revealing a tension between group robustness and poisoning resilience, with an impossibility result for loss-based identification.
Problem
Group robustness methods that heuristically identify and upsample hard-to-learn minority samples, and poisoning defenses that remove outliers, may conflict: the former amplify poisons and the latter remove legitimate minority samples.
Approach
Identifies that difficulty-based heuristics used for pseudo-group annotation cannot separate minority samples from poisons (with an impossibility result under assumptions), and that outlier-based poisoning defenses share the same heuristic and therefore eliminate minority samples along with poisons.
Key results
- Group robustness methods identify poisons as minority at 98.2-100% IDNF, exceeding LRG-1 recall; amplification raises DLBD ASR on CelebA to 97.7% in worst case vs 0.3% ideal.
- EPIc reduces worst-group accuracy by 3.2%-14.3% across datasets and attacks, with the largest drop on CelebA.
- Poisons and LRG-1 samples are eliminated at similar rates by EPIc, while HRG elimination is significantly lower.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| amplify poisons with JTT instead of excluding them from the amplified set (ablation) | JTT · Waterbirds, DLBD | ASR: $ 0.5 \pm 0.3 \%$ -> $ \mathbf{20.4} \pm 5.9 \%$ | Hurt |
| amplify poisons with JTT instead of excluding them from the amplified set (ablation) | JTT · Waterbirds, SA | ASR: $ 0.8 \pm 1.3 \%$ -> $ \mathbf{31.4} \pm 6.0 \%$ | Hurt |
| amplify poisons with JTT instead of excluding them from the amplified set (ablation) | JTT · Waterbirds, GM | ASR: $ 13.3 \pm 11.5 \%$ -> $ \mathbf{20.0} \pm 0.0 \%$ | Hurt |
| amplify poisons with JTT instead of excluding them from the amplified set (ablation) | JTT · CelebA 10% subset, DLBD | ASR: $ 0.3 \pm 0.0 \%$ -> $ \mathbf{97.7} \pm 0.1 \%$ | Hurt |
| amplify poisons with GEORGE instead of excluding them from the amplified set (ablation) | GEORGE · Waterbirds, DLBD | ASR: $ 0.4 \pm 0.0 \%$ -> $ \mathbf{15.8} \pm 3.9 \%$ | Hurt |
| let EPIc eliminate under-represented samples rather than preserve them (ablation) | EPIc · Waterbirds, DLBD | WGA: $ 59.0 \pm 2.5 \%$ -> $ \mathbf{55.8} \pm 6.0 \%$ | Hurt |
| let EPIc eliminate under-represented samples rather than preserve them (ablation) | EPIc · Waterbirds, SA | WGA: $ 59.1 \pm 3.0 \%$ -> $ \mathbf{55.3} \pm 3.2 \%$ | Hurt |
| let EPIc eliminate under-represented samples rather than preserve them (ablation) | EPIc · Waterbirds, GM | WGA: $ 50.2 \pm 2.0 \%$ -> $ \mathbf{43.5} \pm 7.8 \%$ | Hurt |
| let EPIc eliminate under-represented samples rather than preserve them (ablation) | EPIc · CelebA 10% subset, DLBD | WGA: $ 54.8 \pm 3.2 \%$ -> $ \mathbf{40.5} \pm 4.5 \%$ | Hurt |
| use JTT to amplify estimated minority samples | JTT · CelebA, Subpopulation Attack | WGA: JTT (WGA) 81.7\% vs ERM (WGA) 43.3\% | Helped |
| use JTT rather than ERM | JTT · CelebA, Subpopulation Attack | ASR: JTT (ASR) 0.1\% vs ERM (ASR) 0.1\% | Hurt |
| reduce JTT upsampling factor from 100 to 50 (ablation) | JTT · Waterbirds, DLBD | ASR: $ \mathbf{20.9} \pm 7.9 \%$ -> $\mathbf{28.8} \pm 6.6 \%$ | Hurt |
Surprising findings
- Smaller upsampling factors (50x or 20x) make the DLBD attack stronger than 100x, by 8.4% and 46.2% ASR respectively.
- Combining group robustness with poisoning defenses fails to resolve the tension: removing minority samples defeats group robustness, while ignoring poisons leaves high attack success.
Limitations
- Experiments are limited to image benchmarks (Waterbirds, CelebA) with a small number of groups and ResNet models.
- The impossibility result relies on specific assumptions (binary labels and attributes, equal variances, independence).
- Only defenses assuming poisons are hard to learn are studied in depth; other defense categories are discussed but not evaluated.
Experimental setup
Datasets: Waterbirds CelebA MultiNLI Spawrious
Benchmarks: Waterbirds CelebA
Metrics: worst-group accuracy attack success rate test accuracy identification factor elimination factor
Baselines: ERM Just Train Twice GEORGE EPIc STRIP FedAvg Coordinate Median Trimmed Mean SparseFed AFR
Builds on: Just Train Twice GEORGE EPIc Group DRO Subpopulation Attack
Closest papers by meaning
- Lethal Dose Conjecture on Data Poisoning (NeurIPS 2022 · similarity 0.88)
- What Distributions are Robust to Indiscriminate Poisoning Attacks for Linear Learners? (NeurIPS 2023 · similarity 0.88)
- On Robustness of Linear Classifiers to Targeted Data Poisoning (AAAI 2026 · similarity 0.88)
- Multigroup Robustness (ICML 2024 · similarity 0.85)
- Model Poisoning Attacks to Federated Learning via Multi-Round Consistency (CVPR 2025 · similarity 0.84)
- Scaling Trends for Data Poisoning in LLMs (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.