Research map / Adversarial robustness and security / Attacks on text and graphs
Foundation Model-oriented Robustness: Robust Image Model Evaluation with Pretrained Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Foundation Model-oriented Robustness (FMR), which generates perturbed test images with VQGAN guided by the evaluated model's loss and constrained to keep labels via a majority vote of foundation models, then measures accuracy relative to the foundation-model oracle.
Problem
Fixed robustness benchmarks and norm-bounded or predefined perturbations are limited and can lead to overfitting/selection bias, so robustness of image classifiers is hard to evaluate beyond them with a consistent metric.
Approach
Generate dynamic, model-specific perturbed test images via VQGAN latent optimization that maximize the evaluated model's classification loss while a zoo of pretrained foundation models (CLIP, ConvNeXt-T-CvSt, CoCa) majority-vote to keep the original label; define FMR as perturbed accuracy normalized by standard accuracy.
Key results
- On ImageNet, Swin achieves the highest FMR among standard models (67.26) despite lower SA (81.67), while ResNet50 gets 40.11 FMR.
- Among robust ResNet50 variants, DAT achieves FMR 68.66 (PA 52.57) versus 40.11 for vanilla ResNet50, while SIN (40.07) falls below vanilla.
- Swin Transformer head-number ablation shows FMR rising from 64.85 to 69.97 as heads go from (2,4,8,16) to (5,10,20,40), with SA dropping from 80.82 to 81.05 non-monotonically.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace vanilla ResNet50 training with SIN | ResNet50 · ImageNet | FMR: 40.11 -> 40.07 | No clear effect |
| Apply ANT | ResNet50 · ImageNet | FMR: 40.11 -> 40.14 | No clear effect |
| Apply Augmix | ResNet50 · ImageNet | FMR: 40.11 -> 50.68 | Helped |
| Apply DeepAug and Augmix together | ResNet50 · ImageNet | FMR: 40.11 -> 55.56 | Helped |
| Apply DAT | ResNet50 · ImageNet | FMR: 40.11 -> 68.66 | Helped |
| Increase Swin Transformer head configuration from (2,4,8,16) to (5,10,20,40) (ablation) | Swin Transformer · ImageNet | FMR: 64.85 -> 69.97 | Helped |
| Use more heads in Swin Transformer (ablation) | Swin Transformer · ImageNet | standard accuracy: 80.82; 81.98; 81.67; 81.05 | Mixed |
| Replace VQGAN with ADM, Improved DDPM, Efficient-VDVAE, or StyleGAN-XL as image generator (ablation) | evaluation protocol · ImageNet | FMR: 39.71–56.92 | No clear effect |
| Initialize perturbation from FGSM adversarial images (ablation) | ResNet evaluation protocol · CIFAR10 | FMR: 57.80 -> 65.79 | Helped |
| Train with generated augmentation images (ablation) | ResNet · CIFAR10 | FMR on vanilla-generated data: 57.82 -> 89.13 | Helped |
| Train with generated augmentation images (ablation) | ResNet · CIFAR10 | FMR on augmented-model-generated data: 58.03 -> 78.61 | Helped |
| Sparsify VQGAN by masking 99.31% of dimensions (ablation) | VQGAN · 9-class ImageNet and ImageNet | runtime reduction: 12.7%; 28.5% | Helped |
Surprising findings
- Standard-accuracy-oriented ranking differs from FMR ranking: Swin has suboptimal clean accuracy but the best FMR among standard models.
- Corruption-robust methods SIN, ANT, and ANT+SIN, which are tuned for ImageNet-C, show FMR no better than vanilla ResNet50 under the proposed protocol.
Limitations
- Evaluation is scoped by the power and biases of the foundation model zoo (e.g., CLIP zero-shot accuracy, class imbalance).
- Generated images may shift colors and may be less effective in niche domains such as medical imaging.
- Foundation models like CLIP are vulnerable to adversarial attacks, which could undermine label validation.
Experimental setup
Datasets: MNIST CIFAR10 ImageNet ImageNet-C
Benchmarks: MNIST CIFAR10 9-class ImageNet ImageNet
Metrics: Standard Accuracy (SA) Perturbed Accuracy (PA) Foundation Model-oriented Robustness (FMR)
Baselines: ResNet50 ViT DeiT Twins Visformer Swin ConvNeXt SIN ANT ANT+SIN AugMix DeepAugment
Builds on: VQGAN CLIP PGD adversarial training FGSM DeepAugment AugMix
Closest papers by meaning
- ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editing (CVPR 2023 · similarity 0.88)
- LR0.FM: Low-Res Benchmark and Improving robustness for Zero-Shot Classification in Foundation Models (ICLR 2025 · similarity 0.87)
- Prompting is a Double-Edged Sword: Improving Worst-Group Robustness of Foundation Models (ICML 2024 · similarity 0.87)
- LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models (ICML 2025 · similarity 0.86)
- Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies (ICML 2024 · similarity 0.85)
- SURE: SUrvey REcipes for building reliable and robust deep networks (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.