Research map / Architectures and efficiency / Neural architecture search
Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use few-shot partitioned search instead of one-shot search with 11232 subnetworks (ablation) | AutoDistil SuperLM · MRPC | F1: 85.1 -> 91.2 | Helped |
| Use few-shot partitioned search instead of one-shot search with 11232 subnetworks (ablation) | AutoDistil SuperLM · RTE | accuracy: 62.8 -> 71.8 | Helped |
| Replace MLM with self-attention distillation for SuperLM training (ablation) | AutoDistil · MRPC | F1: 89.4 -> 91.2 | Helped |
| Replace MLM with self-attention distillation for SuperLM training (ablation) | AutoDistil · RTE | accuracy: 68.2 -> 71.5 | Helped |
| Continue training the searched students after self-attention SuperLM distillation (ablation) | AutoDistil · MRPC | F1: 91.2 -> 91.0 | No clear effect |
| Continue training the searched students after self-attention SuperLM distillation (ablation) | AutoDistil · RTE | accuracy: 71.5 -> 71.8 | Helped |
| Use alternating layer dropping rather than top-layer dropping (ablation) | AutoDistil subnetworks · RTE | accuracy: 68.5 -> 71.8 | Helped |
| Use alternating layer dropping rather than mixed alternate-then-top dropping (ablation) | AutoDistil subnetworks · MRPC | F1: 85.7 -> 91.2 | Helped |
| Use Wiki+Book instead of MNLI data for SuperLM training (ablation) | AutoDistil · MRPC | F1: 88.3 -> 91.2 | Helped |
| Use Wiki+Book instead of MNLI data for SuperLM training (ablation) | AutoDistil · RTE | accuracy: 65.4 -> 71.8 | Helped |
| Use MNLI as a task-proxy search signal rather than task-agnostic search | AutoDistil SuperLM-small · GLUE (dev), average | average task score: 79.6 -> 79.9 | Helped |
| Use AutoDistil Agnostic instead of MiniLM | BERT-base compression · GLUE (dev), average | FLOPs: 5.66 -> 2.13 | Helped |
Closest papers by meaning
- DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.88)
- einspace: Searching for Neural Architectures from Fundamental Operations (NeurIPS 2024 · similarity 0.88)
- Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions (AAAI 2025 · similarity 0.87)
- TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.87)
- RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy (ICML 2025 · similarity 0.87)
- Insights from the Use of Previously Unseen Neural Architecture Search Datasets (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.