Research map / Image generation / Autoregressive and latent generation
ImageNet-D: Benchmarking Neural Network Robustness on Diffusion Synthetic Object
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds ImageNet-D, a robustness benchmark of 4,835 hard images generated by Stable Diffusion across background, texture and material nuisances, mined via shared failures of surrogate vision models and human-verified.
Problem
Existing synthetic robustness benchmarks (ImageNet-C, ImageNet-9, Stylized-ImageNet) cover limited variations and have low image fidelity, so they do not comprehensively test vision model robustness to diverse nuisances such as backgrounds, textures and materials.
Approach
Use a diffusion model steered by language to generate object images under many backgrounds, textures and materials, then keep only images that cause shared failures across several surrogate classifiers, so the hard set transfers to unseen models; human verification ensures validity.
Key results
- ImageNet-D reduces CLIP ViT-B/16 accuracy from 67.95% on ImageNet to 21.90%, a 46.05-point drop.
- ImageNet-D reduces ResNet101 accuracy by 55.02 points and ViT-L/16 by 59.40 points relative to ImageNet.
- Training ResNet18 on Synthetic-easy diffusion images raises ImageNet-D accuracy from 8.6% to 27.86%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate models on ImageNet-D instead of ImageNet | vision and vision-language models · ImageNet-D Total vs ImageNet | accuracy (%): reducing 55.02%; reducing 59.40%; reducing 46.05%; reducing 29.67%; reducing 16.81% | Hurt |
| train with SIN augmentation (ablation) | ResNet50 · ImageNet-D | accuracy (%): 10.22 -> 8.81 | Hurt |
| train with Augmix augmentation (ablation) | ResNet50 · ImageNet-D | accuracy (%): 10.22 -> 8.75 | Hurt |
| train with ANT augmentation (ablation) | ResNet50 · ImageNet-D | accuracy (%): 10.22 -> 10.09 | Hurt |
| train with DeepAugment (ablation) | ResNet50 · ImageNet-D | accuracy (%): 10.22 -> 9.37 | Hurt |
| train with SIN, Augmix, ANT, or DeepAugment (ablation) | ResNet50 · ImageNet-C | mCE: 81.74 -> 78.45; 81.74 -> 74.7; 81.74 -> 76.74; 81.74 -> 70.31 | Helped |
| add ImageNet training data (ablation) | ResNet18 · ImageNet-D | accuracy (%): 8.6 -> 10.4 | Helped |
| add Synthetic-easy training data (ablation) | ResNet18 · ImageNet-D | accuracy (%): 8.6 -> 27.86 | Helped |
| add Synthetic-easy training data instead of ImageNet training data (ablation) | ResNet18 · ObjectNet | accuracy (%): 24.78 -> 26.12 | Helped |
| pretrain ConvNeXt on ImageNet-22K before ImageNet-1K training (ablation) | ConvNeXt · ImageNet-D Background, Texture, and Material | accuracy | Helped |
| change model architecture from ViT to Swin Transformer or ConvNeXt (ablation) | vision models · ImageNet-D Texture and Material | accuracy | Mixed |
Surprising findings
- Data augmentation methods (SIN, AugMix, ANT, DeepAugment) improve ImageNet-C mCE but do not improve, and sometimes worsen, ImageNet-D accuracy.
- LLaVA-NeXT, though better than LLaVA-1.5 on MMMU, achieves lower accuracy on ImageNet-D.
Limitations
- Generated images can mismatch prompt categories, requiring human-in-the-loop quality control.
- Hard-image mining is tied to a chosen set of surrogate models and the accuracy drop saturates beyond four surrogates.
Experimental setup
Datasets: ImageNet-1K ObjectNet Broden
Benchmarks: ImageNet-D ImageNet ObjectNet ImageNet-9 Stylized-ImageNet ImageNet-C
Metrics: top-1 accuracy mean Corruption Error
Baselines: ImageNet-C ImageNet-9 Stylized-ImageNet ObjectNet AugMix DeepAugment SIN ANT
Builds on: Stable Diffusion ImageNet-C ObjectNet Stylized-ImageNet
Closest papers by meaning
- ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editing (CVPR 2023 · similarity 0.88)
- BLURD: Benchmarking and Learning using a Unified Rendering and Diffusion Model (NeurIPS 2024 · similarity 0.88)
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations (ICLR 2019 · similarity 0.88)
- LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models (ICML 2025 · similarity 0.87)
- LL-Bench: Rethinking Low-Level Vision Evaluation in the Era of Large-Scale Generative Models (NeurIPS 2026 · similarity 0.87)
- FerretNet: Efficient Synthetic Image Detection via Local Pixel Dependencies (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.