Research map / Architectures and efficiency / Neural architecture search
Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discovery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Auto-Prox automatically discovers training-free zero-cost proxies for Vision Transformer architecture search using evolutionary search over expression trees, scored by a joint correlation metric on the new ViT-Bench-101 benchmark.
Problem
Training-free ViT architecture search relies on hand-crafted zero-cost proxies that need expert design and generalize poorly across datasets and tasks.
Approach
Represent zero-cost proxies as expression trees over ViT weight/gradient/activation statistics and primitive operations, evolve them with a joint correlation metric (weighted Kendall's tau across multiple datasets) and an elitism-preserve mutation acceptance rule, then use the discovered proxy to pick ViTs without training.
Key results
- On ImageNet-1K ViT search in AutoFormer space, Auto-Prox reaches 75.6% top-1 with 6.4M params at 0.1 GPU days, versus TF-TAS 75.3% at 0.5 GPU days.
- Auto-Prox achieves Kendall's tau of 25.44% on ImageNet-1K ranking, above TF-TAS (14.52%) and SynFlow (14.81%).
- On PiT search space with distillation, Auto-Prox reaches Kendall 82.07% on CIFAR-100 versus TF-TAS 63.83%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Auto-Prox instead of the strongest reported baseline proxy | ViT architecture search in AutoFormer and PiT spaces · CIFAR-100, Flowers, and Chaoyang (distillation accuracy) | distillation accuracy: 77.83 -> 78.26 (CIFAR-100, AutoFormer); 69.18 -> 69.71 (Flowers, AutoFormer); 77.13 -> 77.26 (CIFAR-100, PiT); 68.21 -> | Helped |
| Use Auto-Prox instead of TF-TAS | ViT ranking in the AutoFormer and PiT search spaces · CIFAR-100, Flowers, and Chaoyang, distillation accuracy ranking | Kendall correlation (%): 35.89$_{\pm 0.26}$ -> 55.67$_{\pm 0.74}$ (CIFAR-100, AutoFormer); 63.90$_{\pm 0.04}$ -> 69.19$_{\pm 2.03}$ (Flowers, Aut | Helped |
| Use Auto-Prox instead of TF-TAS | ImageNet-1K ViT architecture search · ImageNet-1K, AutoFormer search space | top-1 accuracy (%): 75.3 -> 75.6 | Helped |
| Use Auto-Prox instead of Snip | ViT ranking · ImageNet-1K | Kendall correlation (%): 14.61$_{\pm 1.55}$ -> 25.44$_{\pm 0.90}$ | Helped |
| Use Auto-Prox instead of TF-TAS | ViT ranking by vanilla accuracy · CIFAR-100, PiT search space | Spearman correlation (%): 84.79$_{\pm 4.57}$ -> 91.01$_{\pm 2.63}$ | Helped |
| Use Auto-Prox instead of EZNAS-A† | Architecture ranking · CIFAR-10, NAS-Bench-201 | Kendall correlation (%): 64.21 -> 68.49 | Helped |
| Use Auto-Prox instead of SynFlow | Training-free search of larger ViTs · CIFAR-100, AutoFormer-B | distillation accuracy (%): 80.33 -> 80.51 | Helped |
| Add the elitism-preserve strategy to evolutionary proxy search (ablation) | Auto-Prox evolutionary search · Proxy search across datasets | search efficiency and proxy results | Helped |
| Optimize the joint correlation metric across multiple datasets rather than optimizing a single-dataset correlation (ablation) | Auto-Prox proxy discovery · Multiple datasets | ranking correlation | Helped |
Surprising findings
- Several hand-crafted proxies (e.g., NWOT, TENAS) show negative or near-zero correlation on some datasets, indicating poor cross-dataset generalization.
Limitations
- Proxy search and ViT-Bench-101 rely on a limited set of two ViT search spaces (AutoFormer, PiT) and four datasets.
- ImageNet-1K ground truth uses inherited supernet subnet accuracy rather than retrained-from-scratch accuracy.
- Small-dataset results mainly use distillation accuracy with a ResNet56 teacher, which differs from vanilla ViT training.
Experimental setup
Datasets: CIFAR-100 Oxford Flowers Chaoyang ImageNet-1K NAS-Bench-201
Benchmarks: ViT-Bench-101 ImageNet-1K CIFAR-100 Flowers Chaoyang NAS-Bench-201
Metrics: Kendall's tau Spearman Pearson distillation accuracy top-1 accuracy search cost (GPU hours)
Baselines: GraSP SynFlow TENAS NWOT TF-TAS Snip EZNAS-A Random
Builds on: TF-TAS SNIP SynFlow AutoFormer AutoML-Zero
Closest papers by meaning
- Training-Free Transformer Architecture Search (CVPR 2022 · similarity 0.91)
- Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights (NeurIPS 2024 · similarity 0.88)
- L-SWAG: Layer-Sample Wise Activation with Gradients Information for Zero-Shot NAS on Vision Transformers (CVPR 2025 · similarity 0.87)
- ParZC: Parametric Zero-Cost Proxies for Efficient NAS (AAAI 2025 · similarity 0.87)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.86)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.