Research map / Architectures and efficiency / Neural architecture search
Training-Free Transformer Architecture Search
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DSS-indicator to search the AutoFormer tiny architecture instead of one-shot TAS search | ViT architecture search in the AutoFormer search space · ImageNet, tiny | Top-1 accuracy (%): 72.2 -> 75.3 | Helped |
| Use DSS-indicator-guided search instead of AutoFormer search | ViT architecture search in the AutoFormer search space · ImageNet, tiny | GPU Days: 24 -> 0.5 | Helped |
| Use DSS-indicator-guided search rather than random PiT subnet selection | PiT search space · ImageNet, tiny | Top-1 accuracy (%): 69.7 -> 73.7 | Helped |
| Use DSS-indicator-guided search rather than random PiT subnet selection | PiT search space · ImageNet, small | Top-1 accuracy (%): 75.1 -> 80.5 | Helped |
| Evaluate MLP synaptic saliency alone rather than MSA synaptic saliency alone (ablation) | ViT proxy ranking · AutoFormer proxy benchmark | Kendall tau: 0.24 -> 0.47 | Helped |
| Combine MSA synaptic diversity with MLP synaptic saliency (ablation) | ViT proxy ranking · AutoFormer proxy benchmark | Kendall tau: 0.41 -> 0.47 | Helped |
| Use MLP saliency without the MSA diversity component (ablation) | DSS-indicator · AutoFormer proxy benchmark | Kendall tau: 0.47 -> 0.697 | Hurt |
| Replace DSS-indicator with TE-score (ablation) | ViT proxy ranking · AutoFormer inherited subnet benchmark, 5 - 7M parameters | Kendall tau: 0.697 -> -0.039 | Hurt |
| Replace DSS-indicator with NASWOT (ablation) | ViT proxy ranking · AutoFormer inherited subnet benchmark, 5 - 7M parameters | Kendall tau: 0.697 -> 0.378 | Hurt |
| Use a fixed all-ones input to make proxy evaluation independent of random seed (ablation) | DSS-indicator · AutoFormer inherited subnet benchmark, 5 - 7M parameters | Kendall tau: 0.697 across random seed 0, 1, 2, and 3 | Helped |
Closest papers by meaning
- Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discovery (AAAI 2024 · similarity 0.91)
- Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights (NeurIPS 2024 · similarity 0.88)
- GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.86)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.86)
- L-SWAG: Layer-Sample Wise Activation with Gradients Information for Zero-Shot NAS on Vision Transformers (CVPR 2025 · similarity 0.86)
- Real-time Core-Periphery Guided ViT with Smart Data Layout Selection on Mobile Devices (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.