Research map / Architectures and efficiency / Neural architecture search
Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds OoD-ViT-NAS, a benchmark of 3,000 ViT subnets from Autoformer supernets evaluated on 8 OoD datasets, and uses it to study how ViT architecture, ID accuracy, training-free NAS proxies, and embedding dimension relate to OoD generalization.
Problem
Vision Transformer architecture design has been optimized for in-distribution accuracy, while its effect on out-of-distribution generalization, and whether NAS proxies can predict OoD accuracy, remain largely unexplored.
Approach
Create a large OoD evaluation benchmark by sampling 1,000 subnets from each of three Autoformer supernets (Tiny/Small/Base), reusing inherited supernet weights instead of training each candidate, and then analyze ID-OoD correlation, training-free NAS proxies, and architectural attributes.
Key results
- Training-free NAS proxies are largely ineffective for OoD prediction; the best proxy #Flops reaches only about 0.354 Kendall tau with average OoD accuracy, while AutoProx-A reaches 0.3303.
- OoD accuracy ranges across architectures in the same search space average 3.81%/4.86%/2.74% (Tiny/Small/Base), with up to 11.85% on some shifts.
- Increasing embedding dimension (ViT-B-32 to 840 dim) gives 48.28% IN-R OoD accuracy versus 44.33% for the larger ViT-L-32 at lower efficiency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase ViT-B-32 embedding dimension to 840 | ViT-B-32 · ImageNet-R | OoD accuracy: 41.58 -> 48.28 | Helped |
| increase Swin-T embedding dimension to 128 | Swin-T · ImageNet-R | OoD accuracy: 46.22 -> 48.28 | Helped |
| increase embedding dimension to 840 | ViT-B-32 · ImageNet-R (IN-100 trained) | OoD accuracy: 41.58 -> 48.28 | Helped |
| increase number of heads to 128 (ablation) | ViT-B-32 · ImageNet-R (IN-100 trained) | OoD accuracy: 41.58 -> 43.47 | Helped |
| increase depth to 16 (ablation) | ViT-B-32 · ImageNet-R (IN-100 trained) | OoD accuracy: 41.58 -> 37.80 | Hurt |
| increase MLP ratio to 3840 (ablation) | ViT-B-32 · ImageNet-R (IN-100 trained) | OoD accuracy: 41.58 -> 42.61 | Helped |
| use simple #Param or #Flops proxies instead of complex training-free NAS proxies | Training-free NAS proxy ranking of ViT architectures · OoD accuracy across 8 datasets and 3 search spaces | Kendall τ correlation: #Param: 0.3600 $\pm$ 0.2321; #Flops: 0.3537 $\pm$ 0.2327; AutoProx-A: 0.3303 $\pm$ 0.2384 | Helped |
| increase embedding dimension | ViT architectures · OoD generalization across benchmark shifts | OoD accuracy | Helped |
| increase network depth | ViT architectures · OoD generalization | OoD accuracy: 0.19 | Mixed |
| increase MLP ratio | ViT architectures · OoD generalization | OoD accuracy: 0.09 | No clear effect |
| increase number of heads | ViT architectures · OoD generalization | OoD accuracy: 0.07 | No clear effect |
| select architectures using ID accuracy Pareto ranking | ViT architectures · OoD shifts | OoD accuracy | Hurt |
Surprising findings
- Simple proxies #Param and #Flops outperform all nine training-free NAS proxies, including ones designed for ViTs or adversarial robustness, in predicting OoD accuracy.
- Pareto-optimal architectures for ID accuracy generally perform sub-optimally under OoD shift.
Limitations
- Benchmark is restricted to the Autoformer search space and subnets inheriting supernet weights rather than independently trained models.
- Training-free NAS evaluation depends on the cost of building the initial robust benchmark.
Experimental setup
Datasets: ImageNet-1K ImageNet-C ImageNet-P ImageNet-A ImageNet-O ImageNet-R ImageNet-Sketch Stylized ImageNet ImageNet-D IN-100
Benchmarks: ImageNet-C ImageNet-P ImageNet-A ImageNet-O ImageNet-R ImageNet-Sketch Stylized ImageNet ImageNet-D
Metrics: ID accuracy OoD accuracy AUPR Kendall tau correlation
Baselines: Grasp SNIP MeCo CroZe Jacobian DSS AutoProx-A ViT-L-32 Swin-S
Builds on: Autoformer Lottery Ticket Hypothesis Grasp SNIP AutoProx
Closest papers by meaning
- OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization (CVPR 2022 · similarity 0.89)
- Towards Robust Vision Transformers: Path Dependency Analysis and a Simple Two-Stage Adversarial Training (CVPR 2026 · similarity 0.89)
- Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discovery (AAAI 2024 · similarity 0.88)
- Structured Initialization for Vision Transformers (NeurIPS 2025 · similarity 0.87)
- Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex. (NeurIPS 2024 · similarity 0.87)
- GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.