Research map / Label-efficient and robust learning / Test-time adaptation
On the Adversarial Risk of Test Time Adaptation: An Investigation into Realistic Test-Time Data Poisoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a realistic grey-box, online test-time data poisoning attack (RTTDP) using a distilled surrogate model, a feature-consistency regularized in-distribution objective, and two TTA-aware losses (NHE, BLE), showing TTA methods are more robust than previously believed under realistic constraints.
Problem
Existing test-time data poisoning attacks on test-time adaptation assume unrealistic capabilities (white-box access to online weights, access to other users' benign samples, or offline injection of all poisoned data), which may overstate TTA's adversarial risk.
Approach
Attack TTA under a grey-box, online protocol by distilling a surrogate model from the adversary's own queries, replacing benign-sample outer objectives with adversary-only samples plus a KL-based intermediate feature consistency regularizer to enable in-distribution poisoning, and designing high-entropy (NHE) and balanced low-entropy (BLE) objectives tailored to TTA methods.
Key results
- On CIFAR10-C, the proposed NHE attack raises average benign error to 53.65% (uniform) versus 35.93% for MaxCE and 46.91% for TePA, averaged over TTA methods.
- The attacks are substantially weaker on EMA-based and advanced TTA methods like CoTTA and ROID, where poisoning transfer is limited, supporting the claim that TTA is more robust than believed.
- Ablation shows feature consistency regularization improves BLE on CIFAR10-C TENT from 19.26-ish baseline to 54.07% with the surrogate model, and entropy thresholding, data augmentation, and EMA together reduce NHE attack error to about 20%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NHE instead of the strongest listed baseline, TePA, under uniform poisoning | TTDP attack on CIFAR10-C · CIFAR10-C uniform | Avg. Err.: 21.73 -> 36.25 | Helped |
| Use NHE instead of the strongest listed baseline, TePA, under uniform poisoning | TTDP attack on CIFAR100-C · CIFAR100-C uniform | Avg. Err.: 46.91 -> 53.65 | Helped |
| Use NHE instead of the strongest listed baseline, TePA, under uniform poisoning | TTDP attack on ImageNet-C · ImageNet-C uniform | Avg. Err.: 65.72 -> 68.04 | Helped |
| Use NHE instead of the strongest listed baseline, MaxCE, under non-uniform poisoning | TTDP attack on ImageNet-C · ImageNet-C non-uniform | Avg. Err.: 64.96 -> 68.07 | Helped |
| Add feature consistency regularization to BLE using the surrogate model (ablation) | BLE attack · CIFAR10-C | AR: 20.03 -> 42.02 | Helped |
| Add feature consistency regularization to BLE using the surrogate model (ablation) | BLE attack · CIFAR100-C | AR: 34.18 -> 54.83 | Helped |
| Add feature consistency regularization to NHE using the surrogate model (ablation) | NHE attack · CIFAR10-C | AR: 38.07 -> 42.72 | Helped |
| Add feature consistency regularization to NHE using the surrogate model (ablation) | NHE attack · CIFAR100-C | AR: 48.44 -> 62.01 | Helped |
| Add feature consistency regularization to BLE using the source model (ablation) | BLE attack · CIFAR10-C | AR: 19.29 -> 27.74 | Helped |
| Add feature consistency regularization to BLE using the source model (ablation) | BLE attack · CIFAR100-C | AR: 32.67 -> 45.15 | Helped |
| Add feature consistency regularization to NHE using the source model (ablation) | NHE attack · CIFAR10-C | AR: 35.43 -> 26.38 | Hurt |
| Add feature consistency regularization to NHE using the source model (ablation) | NHE attack · CIFAR100-C | AR: 54.54 -> 54.94 | Helped |
Surprising findings
- Under realistic protocol without benign-sample access, poisoned data transfers poorly to benign data for advanced TTA methods such as CoTTA and ROID.
- Stochastic parameter restoration improves robustness alone but does not add benefit when combined with other defenses.
Limitations
- Attacks are evaluated only on image classification with corruption benchmarks (CIFAR10-C, CIFAR100-C, ImageNet-C).
- The surrogate model is distilled from the adversary's own queries and assumes the initial source model is known.
- Feature consistency relies on the assumption that adversary's clean samples and benign samples are drawn from similar distributions.
Experimental setup
Datasets: CIFAR10-C CIFAR100-C ImageNet-C
Benchmarks: CIFAR10-C CIFAR100-C ImageNet-C
Metrics: attack success rate (benign error rate) average error rate average rank
Baselines: DIA TePA MaxCE Unlearnable Examples Adversarial Poisoning
Builds on: DIA TePA Unlearnable Examples Adversarial Poisoning TENT
Closest papers by meaning
- PTTA: Purifying Malicious Samples for Test-Time Model Adaptation (ICML 2025 · similarity 0.89)
- Attack To Defend: Exploiting Adversarial Attacks for Detecting Poisoned Models (CVPR 2024 · similarity 0.87)
- MedBN: Robust Test-Time Adaptation against Malicious Test Samples (CVPR 2024 · similarity 0.87)
- Precise Accuracy / Robustness Tradeoffs in Regression: Case of General Norms (ICML 2024 · similarity 0.87)
- Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors (ICLR 2024 · similarity 0.86)
- Monitoring Risks in Test-Time Adaptation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.