Research map / Architectures and efficiency / Neural architecture search
W-PCA Based Gradient-Free Proxy for Efficient Search of Lightweight Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes W-PCA, a gradient-free zero-shot NAS proxy that multiplies parameter count by the number of FFN principal components needed to reach cumulative variance η, to search lightweight BERT-style language models.
Problem
Searching for lightweight NLU language models is expensive with one-shot or training-based NAS, and existing zero-shot NAS proxies designed for vision give biased or weak rankings on language-model search spaces.
Approach
Use a PCA-based forward-only proxy computed on FFN intermediate activations (count of principal components explaining η=0.99 variance), summed over layers and multiplied by the parameter count, then search with a genetic algorithm and refine the found architecture with knowledge distillation.
Key results
- On the FlexiBERT ranking benchmark, W-PCA reaches Kendall tau 0.526 and Spearman rho 0.698, above the best prior zero-shot proxy Head Confidence (0.306 / 0.364).
- On the GLUE test set, W-PCA-Small (15.6M params) averages 78.3, above EfficientBERT+ (77.9) and EfficientBERT++ (78.0) among lightweight baselines.
- On the GLUE dev set, W-PCA-Small reaches 81.4 average at 0.5 GPU days versus 80.4 for EfficientBERT at 58 GPU days.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| multiply V-PCA by parameter count to form W-PCA (ablation) | zero-shot proxy search on FlexiBERT · FlexiBERT benchmark | Kendall τ: 0.466 -> 0.526 | Helped |
| multiply V-PCA by parameter count to form W-PCA (ablation) | zero-shot proxy search on FlexiBERT · FlexiBERT benchmark | Spearman ρ: 0.677 -> 0.698 | Helped |
| use parameter count alone instead of W-PCA (ablation) | architecture search · GLUE dev | AVG: 81.4 -> 80.3 | Hurt |
| use V-PCA alone instead of multiplying by parameter count (ablation) | architecture search · GLUE dev | AVG: 80.8 -> 81.4 | Hurt |
| use W-PCA-Small instead of EfficientBERT | lightweight model search · GLUE dev | AVG: 80.4 -> 81.4 | Helped |
| use W-PCA-Small instead of EfficientBERT | lightweight model search · GLUE dev | search time: 58 d -> 0.5 d | Helped |
| use W-PCA-Small instead of EfficientBERT++ | lightweight model search · GLUE test | AVG: 78.0 -> 78.3 | Helped |
| use W-PCA-Small instead of EfficientBERT++ | lightweight model search · SQuAD v1.1 dev | F1: 86.5 -> 86.7 | Helped |
| use W-PCA-Small instead of EfficientBERT++ | lightweight model search · SQuAD v2.0 dev | F1: 76.1 -> 76.8 | Helped |
| increase η from 0.9 to 0.99 (ablation) | W-PCA ranking proxy · FlexiBERT benchmark | Kendall τ: 0.526 -> 0.513 | Hurt |
| increase η from 0.9 to 0.999 (ablation) | W-PCA ranking proxy · FlexiBERT benchmark | Kendall τ: 0.526 -> 0.499 | Hurt |
| replace zero-shot search with one-shot supernet training | W-PCA-Tiny search · GLUE test | AVG: 75.9 -> 76.2 | Helped |
Surprising findings
- Multiplying parameter count by the PCA proxy improves ranking correlation and downstream accuracy over either factor alone.
- Architectures found by zero-shot search are randomly initialized before fine-tuning, and the paper reports that one-shot supernet pretraining gives only minor accuracy gains (e.g., 78.4 vs 78.3 for W-PCA-Small).
Limitations
- Evaluated mainly on lightweight models around 10-16M parameters; applicability to larger generative LLMs is only explored in an appendix.
- The proxy relies on a fixed η (0.99) chosen by inspection, and results are sensitive to the η choice across BERT and MobileBERT blocks.
- Zero-shot searched models are trained from random initialization, and the gap to one-shot-supernet-based search is small in the reported experiments.
Experimental setup
Datasets: English Wikipedia BooksCorpus OpenWebText GLUE SQuAD SlimPajama
Benchmarks: GLUE SQuAD v1.1 SQuAD v2.0 FlexiBERT benchmark
Metrics: Kendall tau Spearman rho GLUE average score EM/F1 latency search time
Baselines: Synaptic Diversity Synaptic Saliency Activation Distance Jacobian Cosine Head Confidence Softmax Confidence EfficientBERT EfficientBERT+ TinyBERT-4 MobileBERT-tiny DistilBERT-6 NAS-BERT
Builds on: FlexiBERT SPOS EfficientBERT ELECTRA Synaptic saliency zero-shot proxies
Closest papers by meaning
- NEAR: A Training-Free Pre-Estimator of Machine Learning Model Performance (ICLR 2025 · similarity 0.85)
- Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models (NeurIPS 2024 · similarity 0.85)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.85)
- Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models (NeurIPS 2024 · similarity 0.84)
- Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.84)
- Revisiting Neural Networks for Few-Shot Learning: A Zero-Cost NAS Perspective (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.