Research map / Label-efficient and robust learning / Recognition and long-tail
DCLP: Neural Architecture Predictor with Curriculum Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DCLP, a curriculum-guided contrastive pre-training framework for GIN-based neural architecture performance predictors that uses a difficulty-scheduled order of augmented positives and ranking-based fine-tuning to reach strong NAS results with few labeled architectures.
Problem
Predictor-based NAS needs many fully trained networks to train accurate predictors; unsupervised contrastive pre-training on architecture graphs is unstable and yields poor predictions.
Approach
Schedules contrastive training data by a graph-edit-distance-based difficulty measure with a non-monotonic curriculum (periodic decreases in difficulty) so that the contrastive learner converges smoothly, then fine-tunes a predictor with a ranking (ListMLE) loss or normalized MSE.
Key results
- On NAS-Bench-201, DCLP+RS reaches 94.34% on CIFAR-10 and 73.5% on CIFAR-100 using 50 queries, with 0.004 GPU h search cost.
- On DARTS space CIFAR-10, DCLP-RS achieves 2.48% test error with 4.0 GPU h training and 0.05 GPU h search.
- On DARTS RNN space PTB, DCLP-RS achieves 56.19 test perplexity, better than the listed baselines (best prior 56.94 for CL-V2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add contrastive pre-training (ablation) | DCLP predictor · DARTS space CIFAR-10 | test error (%): 5.36±0.95 -> 2.48±0.02 | Helped |
| add DCLP's curriculum to contrastive learning (ablation) | DCLP predictor · DARTS space CIFAR-10 | test error (%): 5.36±0.95 -> 2.48±0.02 | Helped |
| replace easy-to-hard curriculum with DCLP's fluctuating curriculum (ablation) | DCLP predictor · DARTS space CIFAR-10 | test error (%): 2.66±0.17 -> 2.48±0.02 | Helped |
| use hard-to-easy curriculum instead of no curriculum (ablation) | DCLP contrastive pre-training · DARTS space CIFAR-10 | test error (%) | Hurt |
| add ranking loss (ablation) | DCLP fine-tuning · DARTS space CIFAR-10 | test error (%): 6.42±2.00 -> 5.80±1.75 | Helped |
| replace embedding similarity with structural similarity (WD) (ablation) | DCLP contrastive pre-training · NAS-Bench-101 | accuracy (%): 93.65±0.18 -> 94.17±0.06 | Helped |
| replace GCN encoder with GIN encoder (ablation) | DCLP predictor · NAS-Bench-101 | accuracy (%): 93.98±0.05 -> 94.17±0.06 | Helped |
| replace GAT encoder with GIN encoder (ablation) | DCLP predictor · NAS-Bench-101 | accuracy (%): 93.79±0.08 -> 94.17±0.06 | Helped |
| use DCLP+RS instead of DCNAS | NAS search on NAS-Bench-201 CIFAR-10 · NAS-Bench-201 CIFAR-10 | accuracy (%): 94.29±0.07 -> 94.34±0.03 | Helped |
| use DCLP+RS instead of FreeREA | NAS search on NAS-Bench-201 CIFAR-100 · NAS-Bench-201 CIFAR-100 | accuracy (%): 73.30±0.31 -> 73.5±0.3 | Helped |
| use DCLP+RS instead of DCNAS | NAS search on NAS-Bench-201 ImageNet-16-120 · NAS-Bench-201 ImageNet-16-120 | accuracy (%): 46.41±0.14 -> 46.54±0.03 | Helped |
| use DCLP-RS instead of GMAE | NAS search in DARTS space · DARTS space CIFAR-10 | test error (%): 2.50±0.03 -> 2.48±0.02 | Helped |
Surprising findings
- Learning curriculum from hard to easy performs worse than no curriculum at all.
- Ranking-based fine-tuning losses outperform vanilla MSE, and the authors argue MSE is poorly aligned with ranking accuracy.
Limitations
- Difficulty measure approximates Wasserstein distance with Levenshtein distance, which the authors acknowledge may have limitations.
- Some reported comparisons use baseline numbers across differing settings and training data sizes, which complicates direct comparison.
Experimental setup
Datasets: NAS-Bench-101 NAS-Bench-201 DARTS search space DARTS RNN space PTB CIFAR-10 ImageNet
Benchmarks: NAS-Bench-101 NAS-Bench-201 DARTS space CIFAR-10 DARTS space ImageNet DARTS RNN space PTB
Metrics: Kendall's Tau test accuracy test error test perplexity query count GPU hours
Baselines: NASWOT TENAS HAAP GMAE HOP FreeREA DCNAS ReNAS CTNAS CATE DARTS- CL-V2
Builds on: InfoNCE curriculum learning ListMLE GIN GMAE
Closest papers by meaning
- Contrastive Neural Architecture Search With Neural Architecture Comparators (CVPR 2021 · similarity 0.91)
- Encodings for Prediction-based Neural Architecture Search (ICML 2024 · similarity 0.87)
- A Semi-Supervised Assessor of Neural Architectures (CVPR 2020 · similarity 0.87)
- Insights from the Use of Previously Unseen Neural Architecture Search Datasets (CVPR 2024 · similarity 0.86)
- Neural Architecture Retrieval (ICLR 2024 · similarity 0.86)
- einspace: Searching for Neural Architectures from Fundamental Operations (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.